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Introduction 

This guide is intended to help educators, 
out-of-school time (OST) program pro- 
viders, and school and district adminis- 
trators structure academically focused 
out-of-school time programs. OST is an 
opportunity to supplement learning from 
the school day and provide targeted as- 
sistance to students whose needs extend 
beyond what they can receive in the class- 
room. With an increasing focus on school 
accountability and student performance, 
OST can play a meaningful role in improv- 
ing academic achievement and closing the 
gap between low- and high-performing 
students. Although OST programs oper- 
ate nationwide, disagreement about which 
aspects of these programs are beneficial 
for student achievement remains. This 
practice guide includes concrete recom- 
mendations for structuring an effective 
academically oriented OST program, and it 
illustrates the quality of the evidence that 
supports these recommendations. The 
guide also acknowledges possible imple- 
mentation challenges and suggests solu- 
tions for circumventing the roadblocks. 

A panel of experts in OST programs and 
research methods developed the recom- 
mendations in this guide and determined 
the level of evidence for each recommen- 
dation. The evidence considered in de- 
veloping this guide ranges from rigorous 
evaluations of OST programs to expert 
analyses of practices and strategies in OST. 
In looking for effective practices, the panel 
paid particular attention to high-qual- 
ity experimental and quasi-experimental 
studies, such as those meeting the criteria 
of the What Works Clearinghouse (WWC),^ 
and to patterns of practices that are repli- 
cated across programs. 

As with all WWC practice guides, the rec- 
ommendations in this guide are derived 
from and supported by rigorous evidence. 



1. http://www.whatworks.ed.gov/. 



when possible. The research base for this 
guide was identified through a comprehen- 
sive search for studies evaluating academi- 
cally oriented OST interventions and prac- 
tices. An initial search for research on OST 
programs conducted in the United States 
in the past 20 years (1988-2008) yielded 
more than 1,000 studies. Of these, 130 
studies examined school-based OST pro- 
grams that serve elementary and middle 
school students and were eligible for fur- 
ther review. These studies were reviewed 
by the WWC to determine whether they 
were consistent with WWC standards. Of 
the 130 studies, 22 met WWC standards 
or met the standards with reservations. 
These 22 studies of 18 different OST pro- 
grams represent the strongest evidence of 
the effectiveness of OST programs.^ 

In keeping with the WWC standards for 
determining levels of evidence, the panel 
relied on the following definitions (see 
Table 1): 

A strong rating refers to consistent and 
generalizable evidence that an inter- 
vention strategy or program improves 
outcomes. 3 

A moderate rating refers either to evidence 
from studies that allow strong causal con- 
clusions but cannot be generalized with 
assurance to the population on which a 
recommendation is focused (perhaps be- 
cause the findings have not been widely 
replicated) or to evidence from studies that 
are generalizable but have more causal 
ambiguity than that offered by experimental 
designs (e.g., statistical models of correla- 
tional data or group comparison designs for 



2. See Table D2 for a summary of which studies 
are relevant to each recommendation. 

3. Following WWC guidelines, improved out- 
comes are indicated by either a positive statisti- 
cally significant effect or a positive, substantively 
important effect size (i.e., greater than 0.25). See 
the WWC guidelines at http://ies.ed.gov/ncee/ 
wwc/pdf/wwc_versionl_standards.pdf. 
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which equivalence of the groups at pretest 
is uncertain). 

A low rating refers to expert opinion based 
on reasonable extrapolations from re- 
search and theory on other topics and 
evidence from studies that do not meet 
the standards for moderate or strong 
evidence. 

It is important for the reader to remem- 
ber that the level of evidence rating is not 
a judgment by the panel on how effective 
each of these recommended practices will 
be when implemented, nor is it a judg- 
ment of what prior research has to say 
about their effectiveness. The level of evi- 
dence ratings reflect the panel’s judgment 
of the quality of the existing literature to 
support a causal claim that when these 
practices have been implemented in the 
past, positive effects on student academic 
outcomes were observed. They do not re- 
flect judgments of the relative strength of 
these positive effects or the relative impor- 
tance of the individual recommendations. 
Thus, a low level of evidence rating does 
not indicate that the recommendation is 
any less important than other recommen- 
dations with a strong or moderate rating. 
Rather, it suggests that the panel cannot 
point to a body of research that demon- 
strates its effect on student achievement. 
In some cases, this simply means that the 
recommended practices would be diffi- 
cult to study in a rigorous, experimental 
fashion; in other cases, it means that re- 
searchers have not yet studied this prac- 
tice, or that there is ambiguous evidence 
of effectiveness.^ 

Citations in the text refer to studies of 
programs that have implemented vari- 
ous practices. Not all of these programs 
contribute to the level of evidence rating; 



4. For more information, see the WWC Frequently 
Asked Questions page for practice guides, http:// 
ies.ed.gov/ncee/wwc/references/idocviewer/ 
Doc.aspx?docId=15&tocId=3. 



although some of these programs have 
had rigorous evaluations of their impacts, 
others have not. Furthermore, some of the 
programs that have been rigorously evalu- 
ated have found positive effects on aca- 
demic achievement; others have not.^ 

The What Works Clearinghouse 
standards and their relevance 
to this guide 

In terms of the levels of evidence indi- 
cated in Table 1, the panel relied on WWC 
Evidence Standards to assess the quality of 
evidence supporting educational programs 
and practices. WWC addresses evidence 
for the causal validity of instructional pro- 
grams and practices according to WWC 
standards. Information about these stan- 
dards is available at http://ies.ed.gov/ncee/ 
wwc/pdf/wwc_versionl_standards.pdf The 
technical quality of each study is rated and 
placed into one of three categories: 

• Meets Evidence Standards for random- 
ized controlled trials and regression 
discontinuity studies that provide the 
strongest evidence of causal validity. 

• Meets Evidence Standards with Res- 
ervations for all quasi-experimental 
studies with no design flaws and ran- 
domized controlled trials that have 
problems with randomization, attri- 
tion, or disruption. 

• Does Not Meet Evidence Screens for 
studies that do not provide strong evi- 
dence of causal validity. 

Following the recommendations and sug- 
gestions for carrying out the recommen- 
dations, Appendix D presents more in- 
formation on the research evidence that 
supports each recommendation. 



5. Table D1 summarizes the details and effective- 
ness of studies consulted for the evidence rating 
of this guide. 
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Table 1. Institute of Education Sciences levels of evidence for practice guides 



Strong 


In general, characterization of the evidence for a recommendation as strong requires both 
studies with high internal validity (i.e., studies whose designs can support causal conclu- 
sions) and studies with high external validity (i.e., studies that in total include enough of 
the range of participants and settings on which the recommendation is focused to sup- 
port the conclusion that the results can be generalized to those participants and settings). 
Strong evidence for this practice guide is operationalized as 

• A systematic review of research that generally meets WWC standards (see http://ies. 
ed.gov/ncee/wwc/) and supports the effectiveness of a program, practice, or approach 
with no contradictory evidence of similar quality; OR 

• Several well-designed, randomized controlled trials or well-designed quasi-experi- 
ments that generally meet WWC standards and support the effectiveness of a program, 
practice, or approach, with no contradictory evidence of similar quality; OR 

• One large, well-designed, randomized controlled, multisite trial that meets WWC stan- 
dards and supports the effectiveness of a program, practice, or approach, with no 
contradictory evidence of similar quality; OR 

• For assessments, evidence of reliability and validity that meets the Standards for 
Educational and Psychological Testing.® 


Moderate 


In general, characterization of the evidence for a recommendation as moderate requires 
studies with high internal validity but moderate external validity or studies with high 
external validity but moderate internal validity. In other words, moderate evidence is 
derived from studies that support strong causal conclusions but generalization is uncer- 
tain or studies that support the generality of a relationship but the causality is uncertain. 
Moderate evidence for this practice guide is operationalized as 

• Experiments or quasi-experiments generally meeting WWC standards and supporting 
the effectiveness of a program, practice, or approach with small sample sizes and/ 
or other conditions of implementation or analysis that limit generalizability and no 
contrary evidence; OR 

• Comparison group studies that do not demonstrate equivalence of groups at pretest 
and, therefore, do not meet WWC standards but that (1) consistently show enhanced 
outcomes for participants experiencing a particular program, practice, or approach 
and (2) have no major flaws related to internal validity other than lack of demonstrated 
equivalence at pretest (e.g., only one teacher or one class per condition, unequal 
amounts of instructional time, highly biased outcome measures); OR 

• Correlational research with strong statistical controls for selection bias and for dis- 
cerning influence of endogenous factors and no contrary evidence; OR 

• Eor assessments, evidence of reliability that meets the Standards for Educational and 
Psychological Testing'^ but with evidence of validity from samples not adequately repre- 
sentative of the population on which the recommendation is focused. 


Low 


In general, characterization of the evidence for a recommendation as low means that the 
recommendation is based on expert opinion derived from strong findings or theories in 
related areas and/or expert opinion buttressed by direct evidence that does not rise to 
the moderate or strong level. Low evidence is operationalized as evidence not meeting 
the standards for the moderate or high level. 



a. American Educational Research Association, American Psychological Association, and National Council on Measurement 
in Education (1999). 

b. Ibid. 
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Structuring out-of- 
school time to improve 
academic achievement 

Overview 

Over the past three decades, changing 
labor force patterns in the United States 
have significantly increased the need for 
child care for school-age children. In 2000, 
nearly half of school-age children with 
working mothers spent time in non-pa- 
rental supervised settings when they were 
not in school, including before- and after- 
school programs, family child care homes, 
and the homes of relatives.® Commonly 
known as out-of-school time (OST), this 
period outside of the school day when 
children are not with their parents has re- 
ceived extensive policy attention, focused 
on both the risks of negative influences 
during this time and the potential benefits 
the time holds for the positive develop- 
ment of school-age children. 

Although many OST settings are designed 
primarily to provide a safe place for chil- 
dren to be outside of the traditional school 
day while parents work, there is now a 
broader movement toward using OST to 
bridge the gap between high- and low- 
achieving students and to give students 
more time to learn if they need it.^ Aca- 
demically oriented out-of-school programs 
and services are promising because stu- 
dents spend twice as much of their wak- 
ing hours outside of the classroom as in 
it,® and OST periods, especially summer 
breaks, are the times when the achieve- 
ment gap widens. 9 



6. Capizzano, Tout, and Adams (2000). 

7. Halpern (1999). 

8. Hofferth and Sandberg (2001). 

9. Heyns (1978); Alexander, Entwisle, and Olson 
(2007a, 2007b); Downey, von Hippel, and Broh 
(2004); Cooper et al. (1996). 



OST programs offer a promising approach 
to enhancing students’ academic skills and 
to closing the achievement gap. In recog- 
nition of this promise, funding for OST 
programs and services has grown in the 
past few years. As part of the No Child Left 
Behind (NCLB) legislation, districts are re- 
quired to spend 5 percent to 20 percent 
of all Title 1 funds on supplemental educa- 
tional services (SES).^® Further, some states 
expanded funding for OST programs, even 
in the face of reduced budgets. For exam- 
ple, in 2002, California voters approved 
the addition of approximately a half bil- 
lion dollars annually to existing state after- 
school programs. “ 

Similarly, the number of OST programs, in- 
cluding after-school, weekend, and summer 
programs and SES, has been increasing. In 
1995, a U.S. Census Bureau study of child 
care arrangements showed that 5.6 percent 
of children ages 5 to 14 received care in a 
before- or after-school program accord- 
ing to parents in the sample. In 2005, 20 
percent of K-8 students participated in a 
before- or after-school program.^® 



10. Supplemental educational services (SES) are 
tutoring or other academic support services of- 
fered outside the regular school day, at no charge 
to students or their families, by public or private 
providers that have been approved by the state. 
Districts are required to offer SES to low-income 
students in schools that have fallen short of 
adequate yearly progress (AYP) standards for a 
third time (after missing AYP for two consecutive 
years). Students and their parents are permitted 
to choose among state-approved SES providers, 
which come in all varieties, including national 
for-profit firms, local nonprofits, faith-based 
organizations, institutions of higher education, 
and local school districts (which are permitted 
to become approved providers unless they are 
themselves identified for improvement under No 
Child Left Behind [NCLB]). 

11. Administration for Children & families (n.d.). 
State afterschool profiles: California. Washington, 
DC: U.S. Department of Health and Human Ser- 
vices. Retrieved May 29, 2009, from http://nccic. 
org/afterschool/ca.html. 

12. Smith (2000). 

13. Carver and Iruka (2006). 
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OVERVIEW 



Although it is generally assumed that OST 
programs can provide students with posi- 
tive, academically enriching experiences, 
it is not necessarily known how to struc- 
ture programs to effectively improve stu- 
dent academic outcomes. Although many 
studies lacking comparison groups sug- 
gest that OST programs can benefit stu- 
dents academically,^^ those with more rig- 
orous evaluation designs raise questions 
about these findings. For example, find- 
ings from the national evaluation of the 
21st Century Community Learning Centers 
(CCLC) program, which is the largest after- 
school program in the United States, show 
that, on average, students participating in 
the programs had no improvement in aca- 
demic achievement.^^ 

The evaluation found that 21st CCLC pro- 
grams were not consistently focused on 
academics and often placed more empha- 
sis on sports or extracurricular activities 
because they thought those activities were 
more popular with students and would 
encourage participation in the program.^® 
Students in 21st CCLC programs may not 
have been spending enough time engaged 
in academic content to produce measurable 
gains in achievement. Simply adding time 
to students’ days may not benefit them 



14. Center for Applied Linguistics (1994); Fash- 
ola (1998); Ferreira (2004); Sheldon and Hopkins 
(2008). 

15. U.S. Department of Education (2003). 

16. James-Burdumy, Dynarski, and Deke (2007). 



academically: that time may need to be 
carefully orchestrated to facilitate learn- 
ing and retention of academic material. 
Additionally, the average amount of total 
instructional time received by students in 
a typical OST program may be too low to 
generate meaningful academic effects. 

The findings from the evaluation of En- 
hanced Academic Instruction in After- 
School Programs, sponsored by the Insti- 
tute of Education Sciences (lES), provide 
some evidence for what works in OST in- 
struction.^® The elementary school pro- 
grams delivered school-day math and 
reading curricula adapted to after-school 
settings. Students, who received an av- 
erage of 57 hours of enhanced math in- 
struction (more than the 30-40 hours SES 
students might receive^*!), had modest but 
statistically significant improvements in 
math achievement after one year compared 
with students in a regular after-school pro- 
gram. No differences were found between 
students who received enhanced reading 
instruction and those in a regular after- 
school program. This first year of findings 
provides some indication that instruction 
in OST can improve student achievement 
when delivered in a structured, focused 
format with adequate dosage. 



17. Kane (2004). 

18. Black et al. (2008). 

19. Ibid. 

20. Ross et al. (2008). 

21. Black et al. (2008). 
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Scope of the 
practice guide 

The purpose of this practice guide is to 
provide recommendations for organiz- 
ing and delivering school-based OST pro- 
grams to improve the academic achieve- 
ment of student participants. School-based 
programs include those that are adminis- 
tered by a school or school district, as well 
as programs that are contracted by the 
school or school district and provided by 
other organizations. The panel has limited 
the scope of this practice guide to pro- 
grams that (1) serve elementary and mid- 
dle school students, (2) are organized by 
or conducted in partnership with a school 
or school district, and (3) aim to improve 
academic outcomes. 

The structure and objectives of OST pro- 
grams vary. Some exist as a place where 
students can be safe and occupied for the 
time from school dismissal until parents 
are able to pick them up. Other programs 
are designed to provide social and cultural 
enrichment opportunities or to promote 
healthy outcomes such as grade promotion 
and reduction in risky behaviors. This guide 
targets programs whose primary goal is to 
provide academic instruction to improve 
participants’ achievement.^^ The guide tar- 
gets the following types of programs; 

• after-school or weekend programs 

• summer school 

• SES 

Teachers, principals, district administra- 
tors, and other staff who seek guidance for 
structuring these types of OST programs 



22. In a meta-analysis of 73 after-school pro- 
grams, Durlak and Weissberg (2007) found that 
“the presence of an academic component” was 
the largest predictor of a program producing 
significant academic improvements. 



to improve academic achievement can 
benefit from this guide. State education 
agencies may find the recommendations 
useful for assessing the quality of prospec- 
tive OST programs such as SES providers 
or 21st CCLC programs. Other types of 
programs, such as before-school or non- 
schooTbased programs also may benefit 
from the recommendations in this guide. 
However, these programs were not the 
focus of the panel’s discussions. 

The panel assumed that the basic structure 
of an academically focused OST program 
would include the following components: 

• a place to meet— often this is the school, 
but it also may be a community center 
or other facility 

• regular hours of operation 

• transportation (if necessary) 

• administrative and instructional staff 

• instructional materials or curricula 

Staffing needs will vary by program but 
typically include a program director who 
supervises the operation of the program 
(or program sites if there are multiple loca- 
tions) and manages OST instructors. The 
OST instructors are the front-line staff 
members who interact with students. Ad- 
ditionally, either the OST program or a 
school may employ a coordinator who is 
responsible for maintaining the relation- 
ships among the school, the program site, 
and other partners (see recommendation 
1). School districts that have contracted 
with multiple OST programs may employ 
a district coordinator to monitor all pro- 
grams across schools. 

The evidence base in this guide for the 
effectiveness of OST programs and ser- 
vices on student achievement is based on 
what is known about after-school, summer 
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school, and SES programs targeting el- 
ementary and middle school students in 
low-income, high-needs communities. 
Although many of the recommendations 
may look similar to those for high school 
programs, the objectives of OST programs 
may differ in high school, and the action 
steps and roadblocks will be different 
because of the wider range of activities, 
transportation issues, and instructional 
needs of high school students compared 
with those of younger students. Similarly, 
although the recommendations may be ap- 
plicable to other populations, the literature 
used in this guide largely looks at students 
in urban, low-income, and low-achieving 
schools. Table D1 in Appendix D includes 
details on the populations studied by the 
literature mentioned in this guide. 

The recommendations in this guide can be 
used singly or in combination. The panel 
recommends that readers consider imple- 
menting all recommendations, but any one 
recommendation can be implemented inde- 
pendently. For example, recommendations 
1 and 2 will be most useful to school ad- 
ministrators who want to address low stu- 
dent attendance and ensure that students 
who need academic help will benefit from 
the OST program. Recommendations 3 and 
4 will be useful for teachers who are strug- 
gling with addressing the academic needs 
of their students. Recommendation 5 is 
intended for school, district, and program 
administrators who want to ensure that the 
out-of-school programming offered to stu- 
dents is of high quality and that students 
are benefiting from those services. 

In writing the guide, the panel chose not 
to address the following four areas: 

First, the guide does not address whether 
to provide services after school, before 
school, on weekends, or during the sum- 
mer months. The panel does not believe 
that there is a consensus in the literature 
about how the timing of a program affects 
student outcomes. The panel assumes that 



administrators will make that decision 
based on their available resources and the 
needs and preferences of the communities 
they serve. They may decide that one or a 
combination of these types of programs 
will suit their objectives. 

Second, the guide does not address specific 
instructional practices or teaching strate- 
gies that are unique to reading, math, or 
other content areas. The research base 
that would be required to support con- 
tent-specific recommendations is beyond 
the scope of this guide. For reference, the 
What Works Clearinghouse (WWC) has pub- 
lished a number of practice guides and 
intervention reports devoted to content- 
specific areas such as adolescent literacy, 
beginning reading, and elementary and 
middle school math.^^ 

Third, the guide does not provide details 
on the costs of organizing or operating OST 
programs or of implementing the recom- 
mendations of the panel. The panel rec- 
ognizes that cost is a huge element in the 
decisions that programs make about the 
services they provide, but also that costs 
can vary considerably by factors such as 
the type of program, days of operation, and 
geographical area.^^ Some of the roadblocks 
found at the end of each recommendation 
provide suggestions for ways to minimize 
costs, but the panel directs readers to re- 
cent reports by Public/Private Ventures (P/ 
PV), RAND, and others for more information 
on the costs of OST programs. 

Finally, the guide does not address behav- 
ioral management in the OST context. The 
panel acknowledges that in the OST arena, 
as in the school-day classroom, programs 



23. Institute of Education Sciences, (n.d.). What 
Works Clearinghouse: Practice Guides. Washing- 
ton, DC: U.S. Department of Education. Retrieved 
May 29, 2009, from http://ies.ed.gov/ncee/wwc/ 
publications/practiceguides. 

24. Grossman et al. (2009). 

25. Grossman et al. (2009); Beckett (2008). 
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and practices can impact student behavior. 
In OST, this includes the sites participat- 
ing in the national evaluation of 21st CCLC, 
which were found to have an adverse ef- 
fect on student behavioral outcomes. Al- 
though the panel recognizes that creating 
a safe and orderly environment is a neces- 
sary condition for students to learn, inter- 
ventions that address behavioral manage- 
ment issues were judged to be out of the 
scope of this guide and recommendations 
that are targeted to academic improve- 
ment. The panel directs readers to other 
publications that may be helpful in this 
area, including the WWC practice guide. 
Reducing Behavior Problems in the Elemen- 
tary School Classroom?'^ 

Status of the research 

Overall, the panel believes that the exist- 
ing research on OST practices is not at a 
level to provide conclusive evidence of 
best practices. Studies of OST programs 
tend to examine combined effects of a va- 
riety of practices and procedures on stu- 
dent achievement, making it difficult to 
determine the specific practices contrib- 
uting to achievement gains. Likewise, 
the panel encountered varying impacts 
across OST programs with ostensibly simi- 
lar practices. Many studied interventions 
are practiced on a small scale, necessitat- 
ing a small sample size and often making 
it difficult to find an appropriate compari- 
son group. Low levels of participation or 
attendance, even in large-scale programs, 
also make it difficult to interpret evalua- 
tion results. The panel believes that the 
OST field would benefit from additional, 
rigorous research on OST programs that 
serve a large number of students and have 
achieved high levels of participation. Im- 
proving the research base will provide 



26. James-Burdumy, Dynarski, and Deke (2008). 

27. Epstein et al. (2008). 

28. In these cases, the panel members exercised 
their expert judgment to identify practices likely 
to produce achievement gains for students. 



educators and OST providers alike with 
more definitive information about effec- 
tive practices. 

In offering these recommendations, the 
panel reminds readers that the evidence 
base in support of these recommenda- 
tions, when available, is based on expe- 
riences in OST programs. The relatively 
small amount of literature on academically 
focused programs limited the depth of in- 
formation on effective instructional prac- 
tices in the OST context. To account for 
this, the panel also incorporated relevant 
literature in the broader education field 
and used its expert judgment to identify 
practices that strengthened the OST learn- 
ing environment. 

Summary of the recommendations 

This practice guide offers five recommen- 
dations to improve the ability of OST pro- 
grams to benefit students academically 
(see Table 2). Recommendations 1 and 2 
address how to design an OST program by 
considering its relationship with schools 
and the components that can maximize 
the appeal of the program. Recommenda- 
tions 3 and 4 focus on the delivery of aca- 
demic instruction, and how it can be used 
purposely to improve student engage- 
ment and performance. Recommendation 
5 addresses evaluation of OST programs, 
which is essential for maintaining high 
standards of quality as well as continuous 
improvement of the program design and 
instruction. 

Recommendations 1 and 2 (Design). OST 
programs should include design features 
that ultimately strengthen academic prog- 
ress while fulfilling the needs of parents 
and students. Recommendation 1 empha- 
sizes the importance for OST programs to 
connect with school and classroom activi- 
ties to achieve a shared mission of improv- 
ing academic performance. Further, the 
panel recognizes that sometimes students 
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most likely to benefit from a strong aca- 
demic program may be especially unlikely 
to enroll in or attend OST programs regu- 
larly. Thus, the panel recommends that 
OST programs focus on recruiting and 
retaining targeted students so that they 
receive the dosage necessary to realize 
academic benefits (recommendation 2). 

Recommendations 3 and 4 (Instruction). To 
maximize the educational benefits for stu- 
dents, OST programs should deliver aca- 
demic instruction in a way that responds 
to each student’s needs and engages them 
in learning. Recommendation 3 presents 
strategies for the structuring of instruc- 
tional practices and program content to ad- 
dress the needs of students and effectively 
improve academic outcomes. The recom- 
mendation provides suggestions for orga- 
nizing instructional time in the classroom 
and for facilitating individualized teaching 
by assessing student needs. Recommenda- 
tion 4 encourages OST programs to capital- 
ize on programming flexibility by offering 
activities that students may find especially 



engaging. The panel believes it is particu- 
larly important to engage students when 
they may be fatigued after a long day of 
school; on Saturdays; during the summer 
months; or when they are drawn to partici- 
pate in other, nonacademic activities. To 
avoid the pitfalls of other programs that 
failed to demonstrate positive academic ef- 
fects, the panel suggests that all activities 
have a specific learning objective. 

Recommendation 5 (Evaluation). Finally, 
program improvement depends on the 
articulation of goals and expectations, ef- 
fective management, and the performance 
and experience of staff. Recommenda- 
tion 5 presents strategies for schools and 
districts to use to identify programs that 
are most likely to result in academic im- 
provement and to monitor existing pro- 
grams to ensure that the highest-quality 
services are being provided to students. 
The earlier recommendations in this guide 
should provide programs with a solid 
starting point from which to evaluate an 
OST program. 



Table 2. Recommendations and corresponding levels of evidence 



Recommendation 


Level of evidence 


Design 


1. Align the OST program academically with the school day. 


Low 


2. Maximize student participation and attendance. 


Low 



Instruction 



3. Adapt instruction to individual and small group needs. 


Moderate 


4. Provide engaging learning experiences. 


Low 


Evaluation 


5. Assess program performance and use the results to improve the qual- 
ity of the program. 


Low 
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Checklist for carrying out the 
recommendations 

Recommendation 1. 

Align the OST program academically 
with the school day. 

□ Use OST program coordinators to de- 
velop relationships and maintain ongoing 
communication between schools and the 
OST program. 

CH Designate a school staff person to 
coordinate communication with OST pro- 
grams and help them support school 
needs. 

□ Connect OST instruction to school 
instruction by identifying school-based 
goals and learning objectives. 

CH Coordinate with the school to identify 
staff for OST programs. 

Recommendation 2. 

Maximize student participation 
and attendance. 

□ Design program features to meet the 
needs and preferences of students and 
parents. 

□ Promote awareness of the OST pro- 
gram within schools and to parents. 

□ Use attendance data to identify stu- 
dents facing difficulties in attending the 
program. 



Recommendation 3. 

Adapt instruction to individual 
and small group needs. 

□ Use formal and informal assessment 
data to inform academic instruction. 

CH Use one-on-one tutoring if possi- 
ble; otherwise, break students into small 
groups. 

CH Provide professional development 
and ongoing instructional support to all 
instructors. 

Recommendation 4. 

Provide engaging learning 
experiences. 

CH Make learning relevant by incorpo- 
rating practical examples and connect- 
ing instruction to student interests and 
experiences. 

□ Make learning active through op- 
portunities for collaborative learning and 
hands-on academic activities. 

□ Build adult-student relationships 
among OST program participants. 

Recommendation 5. 

Assess program performance and use 
the results to improve the quality of 
the program. 

CH Develop an evaluation plan. 

CH Collect program and student perfor- 
mance data. 

CH Analyze the data and use findings for 
program improvement. 

CH Conduct a summative evaluation. 
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Recommendatioii 1. 
Align the OST program 
academically with 
the school day 

The panel believes that academic 
alignment with the school day is 
necessary for OST programs to improve 
academic performance. OST programs 
and schools have the shared mission of 
helping students achieve success, and 
collaboration between the two can be 
mutually beneficial. Although alignment 
requires additional effort from staff, 
teachers and principals in schools with 
existing OST programs have voiced 
support for this sort of collaboration. 



An OST program coordinator can 
ensure alignment through regular 
communication with school staff, 
and schools can help by designating 
a school-based coordinator to work 
with the OST coordinator. This sort 
of cooperation helps OST programs 
evaluate their students’ needs and 
provide the most effective instruction 
and services. Both the program and 
the school-based coordinators should 
work to align the instructional activities 
of the OST program with state and 
local content standards, the school 
curriculum, and district- and/or school- 
based learning initiatives. 

Level of evidence: Low 

The level of evidence for this recommen- 
dation is low. There is no direct evidence 
that practices outlined in this recommen- 
dation contribute to improved academic 



29. Goldschmidt, Huang, and Chinen (2007); Bis- 
sell et al. (2002). 

30. Borman and Dowling (2006); Langberg et al. 
(2006); Roderick, Jacob, and Bryk (2004); Borman 
(1997). 



outcomes. Although it was common for 
programs to include some components of 
the panel’s recommendations, none tested 
the effectiveness of this recommendation 
individually, only in combination with the 
other components of OST programs. 

In the panel’s opinion, collaboration can 
improve academic outcomes and in the 
studies reviewed for this guide, two in- 
dependent evaluators recommended that 
collaboration between schools and OST 
programs be strengthened if possible. 
However, we acknowledge that more re- 
search is required to demonstrate the ef- 
fects of stronger alignment. 

Brief summary of evidence to 
support the recommendation 

Fifteen OST programs endeavored to 
collaborate with school-based staff or 
initiatives,^^ but, in general, these efforts 
were not core components of the pro- 
grams. Three programs also expressed 
difficulty or reluctance to coordinate more 
fully.^^ Of the 11 programs with studies 



31. Schacter and Jo (2005) and Center for Applied 
Linguistics (1994) suggested the use of more col- 
laboration when appropriate. 

32. Challenging Horizons Program (CHP) — Lang- 
berg et al. (2006); Early Risers — August et al. 
(2001); Enhanced Academic Instruction — Black et 
al. (2008); Teach Baltimore — Borman and Dowl- 
ing (2006); Chicago Summer Bridge — ^Jacob and 
Lefgren (2004); Los Angeles’s Better Educated Stu- 
dents for Tomorrow (L.A.’s BEST) — Goldschmidt, 
Huang, and Chinen (2007); Youth Services — Child 
Care, Academic Assistance, Recreation, and En- 
richment (YS-CARE) — Bissell et al. (2002); 21st 
CCLC — U.S. Department of Education (2003); Leap 
Frog — McKinney (1995); Nurturing Development 
Partnerships (NDP) — Udell (2003); SES — McKay et 
al. (2008); SES — Ross et al. (2008); SES — Munoz, 
Potter, and Ross (2008); Title I supplementary 
education — Borman (1997); The After-School 
Corporation (TASC) — Relsner et al. (2004); Project 
Adelante — Center for Applied Linguistics (1994); 
After-school tutoring — Leslie (1998). 

33. In James-Burdumy et al. (2005), the authors 
noted that 21st CCLC programs struggled to effec- 
tively coordinate homework help with the school; 
and in U.S. Department of Education (2003), 21st 
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that met WWC standards with or with- 
out reservations,^'* three programs docu- 
mented practices that closely corresponded 
to the panel’s recommendations.^^ In two 
of these, coordination between school- 
teachers and OST instructors was frequent 
and structured. Content and skills taught 
during OST were intentionally designed to 
support students during their school-day 
instruction. One program showed positive 
academic effects, and the other did not.^® 
The purpose of the third program, which 
was a summer school, was to help stu- 
dents achieve proficiency on state exami- 
nations they had not mastered during the 
school year. The curriculum was designed 
by the district with the express purpose 
of helping students meet state standards 
and was closely linked to that goal, but 
by nature of its being a summer school 
program, coordination with individual 
teachers was limited. The evaluation of the 



CCLC programs were found to be supportive 
but not “integrated” (p. 39) with the school, in 
Project Adelante (Center for Applied Linguistics 
1994), program directors recommended closer 
coordination with school-day staff to collect data 
and share information on student progress but 
were concerned about the appropriateness of the 
school-day curriculum for the students that their 
program served. Similarly, Morris, Shaw, and Per- 
ney (1990) expressed reluctance to align their 
after-school tutoring program, Howard Street 
Tutoring, to the school curriculum given that 
their students' classroom instruction was often 
beyond the students’ current reading levels. 

34. CHP — Langberg et al. (2006); Early Risers — 
August et al. (2001); Enhanced Academic Instruc- 
tion — Black et al. (2008); Teach Baltimore — Borman 
and Dowling (2006); Chicago Summer Bridge— Jacob 
and Eefgren (2004); L.A.’s BEST — Goldschmidt, 
Huang, and Chinen (2007); YS-CARE — Bissell et al. 
(2002); 21st CCEC — U.S. Department of Education 
(2003); Eeap Erog — McKinney (1995); NDP — Udell 
(2003); SES— McKay et ah (2008); SES— Ross et al. 
(2008); SES — Munoz, Potter, and Ross (2008). 

35. CHP — Langberg et al. (2006); Chicago Sum- 
mer Bridge — Jacob and Lefgren (2004); Leap 
Prog — McKinney (1995). 

36. CHP — Langberg et al. (2006); Leap Prog — 
McKinney (1995). 

37. CHP — Langberg et al. (2006). 

38. Leap Prog — McKinney (1995). 



program found significant and persistent 
effects on both math and reading for 3rd 
graders but not for 6th graders.^® 

The remaining eight programs included 
some components similar to this recom- 
mendation, but studies indicated that the 
degree of coordination was lower or not 
enough information was provided to de- 
termine the level of alignment between 
programs.^** Of these, one showed positive 
effects,^* two showed mixed effects,^^ and 
five programs showed no detectable aca- 
demic effects. Although more of these 
programs failed to demonstrate effective- 
ness, Table D2 shows that there were seven 
effective programs that did not attempt co- 
ordination with schools. Given the absence 
of a clear pattern of effectiveness based on 
the level of coordination with schools and 
the small sample of programs with high 
levels of coordination, the panel decided 
that the level of evidence was low. 



39. Chicago Summer Bridge — ^Jacob and Eefgren 
(2004). 

40. Early Risers — August et al. (2001); Enhanced 
Academic Instruction — Black et al. (2008); Teach 
Baltimore — Borman and Dowling (2006); E.A.’s 
BEST — Goldschmidt, Huang, and Chinen (2007); 
YS-CARE— Bissell et al. (2002); 21st CCEC— U.S. 
Department of Education (2003); NDP — Udell 
(2003); SES— McKay et al. (2008); SES— Ross et al. 
(2008); SES — Munoz, Potter, and Ross (2008). 

41. Early Risers — August et al. (2001). 

42. Enhanced Academic Instruction — Black et 
al. (2008); Teach Baltimore — Borman and Dowl- 
ing (2006). 

43. E.A.’s BEST— Goldschmidt, Huang, and Chinen 
(2007); YS-CARE— Bissell et al. (2002); 21st CCLC— 
U.S. Department of Education (2003); NDP — Udell 
(2003); SES— Ross et al. (2008); SES— McKay et al. 
(2008); SES — Munoz, Potter, and Ross (2008). 
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How to carry out this 
recommendation 



1 . Use OST program coordinators to develop 
relationships and maintain ongoing com- 
munication between schools and the OST 
program. 

An OST program coordinator can play 
a critical role in ensuring that instruc- 
tional components of an OST program are 
aligned with the school day. Coordinators 
should work directly with teachers and 
administrators from the school to obtain 
information that can be used to guide in- 
struction in the OST program. This can 
be accomplished through regular com- 
munication with key school staff, and also 
through participation in school meetings 
and committees. For example, coordina- 
tors can attend staff meetings, participate 
in common planning periods, serve on 
school leadership teams, and participate 
in parent-teacher organizations. OST co- 
ordinators also can promote the OST pro- 
gram to staff and families by posting in- 
formation about the program on bulletin 
boards or holding OST events during the 
school day to expose other students to the 
OST program. 

When possible, the OST coordinator should 
be housed within the school, spending 
time during daily school hours to be visi- 
ble to both students and teachers. The OST 
coordinator can use these types of oppor- 
tunities to maintain an open relationship 
with teachers, principals, and counselors, 
advocating for and gathering data about 
OST students as necessary.'^'* 

The OST coordinator can take key steps to 
facilitate regular communication between 
OST instructors and classroom teachers that 



44. Center for Applied Linguistics (1994). 



will reinforce and complement the school 
curriculum.'*^ Regular communication can 
help identify the needs and strengths of in- 
dividual students and those strategies that 
are most effective in raising achievement. 
Some examples of steps follow; 

• The OST coordinator can develop a log- 
book that students carry back and forth 
daily to share information about OST ac- 
tivities with classroom teachers.'*® The 
logbook can contain information from 
classroom teachers about homework 
assignments, concepts the student is 
struggling with during the school day, 
or the instructional strategies that are 
most effective with the student.'*'' The 
coordinator should work closely with 
school staff in developing this logbook. 
For a sample logbook, see Exhibit 1.^® 

• The OST coordinator can arrange for 
OST instructors to periodically attend 
common planning periods with class- 
room teachers to align programming 
or collaborate with effective teachers 
to identify best practices and materi- 
als for meeting curriculum goals and 
raising student achievement.'*® 

• The OST coordinator can collaborate 
with school-based staff to identify rele- 
vant professional development that OST 
instructors can attend with schoolteach- 
ers to align instructional strategies and 
to provide funding when possible. 



45. Leslie (1998); U.S. Department of Education 
(2003). 

46. Morris, Shaw, and Perney (1990). 

47. Langberg et al. (2006). 

48. For other examples of logbooks, see SEDL 
National Center for Quality Afterschool (n.d.) and 
Region VII After School Programs (n.d.). 

49. Bott (2006) described the use of this strategy 
in the Gardner Extended Services School; U.S. 
Department of Education (2003). 
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Exhibit 1. Sample logbook 



Information from the Classroom Teacher 


Today’s Date 




Student attended class? 


□ Yes □ No □ Tardy 


By subject: 


Subject 


Subject 2 


Subject 3 


Subject 4 


Topics covered 
in class today 










Today’s homework 
assignment 










Areas in which 
the student needs 
additional help 










Instructional strategies 
that were useful 










Behavior or discipline 
issues 










Other comments 











Information from the OST Instructor 


Today’s Date 




Student attended 
OST program? 


□ Yes □ No □ Tardy 


By subject: 


Subject 1 


Subject 2 


Subject 3 


Subject 4 


Percentage of 
homework completed 
during OST 










Homework items that 
were challenging for 
the student 










Topics covered during 
OST instruction 










Instructional strategies 
or activities that were 
useful 










Behavior or discipline 
issues 










Other comments 











a. This form can be used for single subject classrooms or elementary classrooms. 
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2. Designate a school staff person to coordi- 
nate communication with OST programs and 
help them support school needs. 

Schools can designate a staff member (the 
school-based coordinator) to work with the 
OST program coordinator (or with OST co- 
ordinators from multiple OST programs, 
if relevant). The panel believes that when 
OST programs are well aligned to school- 
day goals and instruction, they can support 
the school in raising student achievement. 
A school-based coordinator can serve as a 
first point of contact in the school for OST 
programs and can help ensure that OST in- 
struction is well aligned with school goals. 
This person will play an important role 
in program-school relations, but it is not 
imperative that the position is full time, 
and an existing teacher or counselor may 
be suited to this role. Key functions of the 
school-based staff designee include 

• Developing a set of standard operat- 
ing procedures for distributing student 
data to OST program staff. Relevant 
data can include results from district- 
or state wide standardized testing, 
student progress reports from teach- 
ers, or even brief informal comments 
from teachers on student strengths 
and weaknesses. 

• Preparing relevant information on 
school operations, academic standards, 
improvement plans, and curricula. 

• Observing and communicating with OST 
staff to identify opportunities for greater 
coordination with the school day. 

It also can be beneficial for school districts 
to designate a key staff person to work with 
OST coordinators. The district designee can 
provide OST coordinators with information 
on standards and curricula. The district 
designee also can play a key role in coor- 
dinating and supervising the activities of 
multiple SES and other OST providers. 



3. Connect OST instruction to school instruc- 
tion by identifying school-based goals and 
learning objectives. 

Information gathered from the school and 
the district can be used to prioritize efforts 
to raise academic achievement and sup- 
port student learning during the school 
day. OST programs should align activi- 
ties, instruction, and any formal curricu- 
lum with the state and local standards, as 
well as the content and curriculum of the 
school day.^° State and local curriculum 
standards are often available online, but 
school and district officials also should di- 
rect OST staff to relevant resources such 
as school improvement plans or other spe- 
cific school- or district-based objectives. 
The OST program need not repeat class- 
room instruction, but it can use different 
methods to support and reinforce what 
students learn in school. 

OST programs can help students develop 
skills that support classroom instruction, 
such as learning how to plan, take notes, 
develop an outline, or study for an upcom- 
ing test. For example, OST providers could 
explicitly teach a skill and require students 
to practice that skill using an assignment 
from the school day. They should follow 
up by checking assignments and confer- 
encing with the classroom teacher. When 
promoting the use of skills from OST dur- 
ing the school day, instructors should be 
careful to coordinate with the classroom 
teacher first to ensure that the relevant 
skill will align with classroom instruction 
and will not disrupt the teacher’s routine. 
This can be useful for helping students de- 
velop note-taking, planning, and studying 
skills that will help them achieve success 
with the school-day curriculum. 



50. Borman and Dowling (2006); Langberg et al. 
(2006); Roderick, Jacob, and Bryk (2004); Borman 
(1997); Udell (2003). 

51. Langberg et al. (2006). 
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Field trips or cultural activities that are 
part of the OST program should be ex- 
plicitly linked to school content and state 
standards. The panel believes that these 
activities need to connect to something 
the students are learning in school to help 
them see how what they learn in school 
relates to their real-life experiences. The 
result can maximize the gains from both 
the OST program and the school day and 
make academic content more relevant to 
students’ lives (see recommendation 4 for 
information on connecting engaging in- 
struction to academic content). 



4. Coordinate with the school to identify 
staff for OST programs. 

OST programs have several roles for which 
effective classroom teachers are well 
suited. The panel recommends that pro- 
grams evaluate how classroom teachers 
can be useful to their programs and hire 
them when appropriate to meet program 
goals. For example, teachers can serve as 
OST coordinators, particularly for sum- 
mer programs in which teachers might 
not face conflicting demands on their time 
from their regular teaching schedules. 
When funding is available to hire effective 
teachers from the school to serve as OST 
instructors, these teachers can use their 
experience and knowledge of instructional 
methods to maximize academic gains for 
participating students. Finally, teachers 
can use their experience to advise and 
mentor less-experienced OST instructors 
or volunteers, especially when budgets are 



52. In 21st CCLC programs, about 67 percent of 
coordinators had experience as classroom teach- 
ers, and 34 percent were currently school-day 
teachers (U.S. Department of Education 2003, 
p. 36). 

53. In Chicago Summer Bridge (Jacob and Lef- 
gren 2004), 21st CCLC (U.S. Department of Educa- 
tion 2003), and Enhanced Academic Instruction 
(Black et al. 2008), for example, schoolteachers 
frequently served as OST instructors. 



tight or sufficient numbers of experienced 
teachers are not available.^'* 

Although little is known about the methods 
or characteristics that define effective teach- 
ers, researchers have discovered that some 
teachers are much better than others at help- 
ing students achieve significant achievement 
gains during the school day.^^ For direct 
instruction or supervisory roles, the panel 
recommends hiring classroom teachers who 
demonstrate success during the school day, 
and the school can support these efforts. To 
identify effective teachers to employ as the 
OST coordinator or as an OST instructor, 
OST programs can seek out award-winning 
teachers or work with administrators to 
identify effective teachers.^® 

Potential roadblocks and solutions 

Roadblock 1.1. The principal does not 
have time to coordinate with OST staff. 

Suggested Approach. OST programs sup- 
port the school by providing additional 
academic assistance to students. To max- 
imize their effectiveness, the panel sug- 
gests that the principal designate someone 
at the school to be responsible for com- 
munications regarding OST programming. 
The OST program can help by clearly com- 
municating the benefits of collaboration to 
the principal. 

Roadblock 1.2. The OST program does 
not have enough money to hire a program 



54. In KindergARTen (Borman, Goertz, and Dowl- 
ing 2008), Teach Baltimore (Borman and Dowling 
2006), NDP (Udell 2003), and Howard Street Tutor- 
ing (Morris, Shaw, and Perney 1990), schoolteachers 
served as advisors to less-experienced instructors 
or volunteers. 

55. See, for example, Hanushek (1992). 

56. Roderick, Jacob, and Bryk (2004). Research- 
ers have demonstrated that principals are good at 
identifying their best and worst teachers, but they 
are not as good at distinguishing among those 
who fall in between (Jacob and Lefgren 2008). 
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coordinator, or the school cannot afford to 
hire a school-based coordinator. 

Suggested Approach. Depending on size 
and scope of the program, a full-time pro- 
gram coordinator might not be necessary. 
The panel believes that programs can be 
successful with a part-time coordinator, 
particularly when the program is small 
or works with only a few students. What 
is important is that both the program and 
the school designate a person who is re- 
sponsible for managing the coordination 
between program and school staff and 
that this person’s role in maintaining com- 
munication is clearly established. Another 
possibility is for a volunteer to take on the 
coordinator role. 

Roadblock 1.3. It is hard to get high-qual- 
ity teachers to work after school because 
of their commitments and responsibilities 
during the school day. 

Suggested Approach. OST programs can 
consider devising flexible staff schedules 
that allow teachers to work for shorter 
periods (e.g., tutoring small groups of 
students for one hour). Programs also 
can consider involving busy but qualified 
teachers in important support roles, such 
as coaching less-experienced instructors 
or providing instructional training. 



Roadblock 1.4. The OST instructor has some 
concerns about aligning instruction with the 
school day, or the classroom teacher has 
some concerns about the OST instruction. 

Suggested Approach. Concerns about 
alignment may signal a need for greater 
communication between the two programs. 
Use the OST coordinator and the school- 
based coordinator to communicate con- 
cerns and ensure that students receive 
the instruction that will benefit them the 
most. This may mean sharing information 
about a particular student’s progress, for 
example. 

Roadblock 1.5. Privacy concerns prohibit 
the transfer of data from school to OST 
program. 

Suggested Approach. OST programs can 
establish a secure system for transferring 
data that meets the security concerns of 
the school and can ensure that only a lim- 
ited number of staff members have access 
to the data. Programs should get formal 
written consent from parents and students 
for their data to be released. OST staff also 
can schedule meetings with teachers and 
staff to gather informal information on 
student performance in school. 
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Recommendatioii 2. 
Maximize student 
participation and 
attendance 

To attract and retain participants, 

OST programs should determine 
which factors prevent students from 
participating in the program and work 
with schools and parents to ensure that 
the program is addressing those factors. 
Parents are critical to this process 
because they are co-decisionmakers 
about students’ participation in OST 
programs, and children generally value 
their parents’ judgment about which 
programs may be beneficial to them.^^ 
Important factors include location, 
transportation, timing, length, program 
offerings, and frequency of services. 
Researchers have found that student 
participation is affected by issues of 
access and convenience, as well as 
by the adequacy and attractiveness 
of the services and features provided 
in the program.^® 



The challenge for OST program 
organizers is to design program 
features to minimize the barriers to 
participation, especially for the students 
most in need of program services and 
most likely to benefit from them. OST 
programs also can increase awareness 
and acceptance by promoting the 
program among school staff and 
families. Greater awareness is likely to 
facilitate communication with schools 
and parents regarding students who 
struggle to attend and could help 
in actively persuading students to 
participate in program activities. 



57. Duffett et al. (2004). 
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Level of evidence: Low 

The panel judged the level of evidence 
supporting this recommendation to be low, 
because there is no conclusive evidence 
that following the action steps in this rec- 
ommendation will lead to higher atten- 
dance or increased academic achievement. 
Given the voluntary nature of most OST 
programs and other barriers discussed in 
this recommendation, regular attendance 
appears to be a difficult goal for many pro- 
grams to reach. Some programs have de- 
voted considerable resources and seem to 
have made efforts to implement the action 
steps described in this recommendation 
and still have trouble getting students to 
attend regularly (see Table D2). However, 
given that attendance is a precursor to an 
OST program’s promoting student learn- 
ing, the panel believes it is particularly 
important for programs to enhance their 
efforts to get students in the door. 

Brief summary of evidence to 
support the recommendation 

The importance of emphasizing partici- 
pation has been pointed out by many ex- 
perts in OST.^^ Although it seems logical 
that students need to attend to receive the 
benefits of a program, there is no rigorous 
evidence demonstrating that the steps rec- 
ommended here will lead to increased par- 
ticipation, and limited evidence that aca- 
demic achievement is increased through 
more exposure to OST programs. A meta- 
analysis of 53 OST programs by Lauer et 
al. (2004) found larger effect sizes in both 
math and reading for programs that con- 
sisted of at least 45 hours of program- 
ming.®*’ The panel believes that if a pro- 
gram is aligned academically with the 



59. Cooper et al. (2000); Granger and Kane (2004); 
Lauver, Little, and Weiss (2004). 

60. The meta-analysis also found that, on average, 
programs with very high durations (more than 100 
hours for math and 210 hours for reading) did not 
have effects significantly different from zero. 
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school day (recommendation 1), adapts 
instruction to individuals and groups (rec- 
ommendation 3), and provides engaging 
learning experiences (recommendation 4), 
greater exposure to that program will yield 
higher academic achievement. 

Since a student’s actual program atten- 
dance, as a percentage of hours of pro- 
gramming offered, may be correlated with 
unobservable characteristics such as mo- 
tivation or family circumstances, it is not 
advisable to draw causal conclusions from 
most studies on the relationship between 
attendance and outcomes. Four evalua- 
tions met WWC standards with or with- 
out reservations for their impact studies 
and also looked at the possible relation 
between level of program attendance and 
academic achievement.®^ Only one found a 
positive correlation between higher atten- 
dance and greater program effects.®^ 

The other evidence for this recommen- 
dation is less direct. Fourteen programs 
used practices similar to the action steps 
recommended by the panel (such as 
using teachers to recruit students, locat- 
ing within schools, offering snacks, and 
including enrichment activities) and also 
had evaluations that met WWC standards 
with or without reservations.®^ Of the 14 
programs, 6 reported some information 



61. Teach Baltimore — Borman and Dowling (2006); 
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and Jo (2005); KindergARTen — Borman, Goetz, 
and Dowling (2008); Early Risers — August et al. 
(2001); Enhanced Academic Instruction — Black et 
al. (2008); Teach Baltimore — Borman and Dowl- 
ing (2006); E.A.’s BEST— Goldschmidt, Huang, and 
Chinen (2007); YS-CARE— Bissell et al. (2002); 21st 
CCLC — U.S. Department of Education (2003); Leap 
Erog— McKinney (1995); NDP— Udell (2003). 



on attendance rates. ®'^ Even when pro- 
grams report attendance, the panel cannot 
isolate which, if any, components of the 
program affected attendance, forcing the 
panel to use its judgment regarding which 
practices contributed to increased atten- 
dance and improved academic outcomes. 
In terms of overall academic effects, 7 of 
the 14 showed positive effects,®® 2 showed 
mixed effects,®® and 5 others showed no 
effects.®^ Despite the lack of consistent 
evidence linking the panel’s suggestions 
to increased academic achievement, the 
panel believes these recommendations, 
faithfully implemented and taking into 
consideration the unique constraints and 
student populations of each program, can 
increase student attendance and, there- 
fore, contribute to achievement gains. 

Only one study provided direct evidence 
on increasing attendance. Black et al. 
(2008) randomly assigned students to ei- 
ther a less-structured, business-as-usual 
after-school program or an enhanced math 
or reading program that included moni- 
toring and incentive systems to increase 
attendance. Students in the enhanced pro- 
gram attended significantly more days 
than did control students. 
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How to carry out this 
recommendation 



1. Design program features to meet the needs 
and preferences of students and parents. 

The panel recommends that the OST pro- 
gram gather information about parent 
preferences with a survey or seek the 
advice of school staff in identifying the 
needs of parents and students. The survey 
could be distributed to parents through 
the school and could ask a short series of 
questions, such as 

• Do you prefer an after-school/summer 
program that is located at your child’s 
school or one that is located at a com- 
munity center? 

• Would you be able to transport your 
child from school to an after-school 
program? 

• What times would you like the pro- 
gram to start and end? 

• Which academic subjects does your 
child need additional support in? 

• How many days per week would you 
send your child to an after-school/ 
summer program? 

The program could consider a similar short 
survey to gauge the preferences of stu- 
dents, especially when serving the higher 
grade levels in which students are more 
likely to choose a program on their own. 

The responses should guide how the pro- 
gram organizes and provides its services. 
This includes working with schools and dis- 
tricts to ensure that design features make 
the program accessible. For example, the 
OST program should consider a location 
that is well situated and easy to get to, 
whether it is a school, community center, or 



place of worship.®® Parents often prefer the 
use of school facilities for services, which 
eliminates the need to move from school 
to another location after school. If the 
program is not located at the school, or if 
the program is serving students from mul- 
tiple schools, schools and districts should 
ensure that transportation to and from the 
program is readily available and affordable 
(or provided at no cost), and that adult su- 
pervision is provided while transporting 
students.^® The panel believes that OST 
programs also should try to operate dur- 
ing hours that are convenient for families, 
particularly for working parents. 

Program features should reflect the con- 
tent that students and parents want, both 
in academic and nonacademic areas. Par- 
ents are likely to be looking for academic 
or homework help that reflects the areas 
in which their children need additional 
help.^^ Since academically oriented OST 
programs may be competing with other 
recreational activities in the same time 
period, programs should offer enrich- 
ment and recreational activities in addi- 
tion to academic instruction. These ac- 
tivities can include theater, music, arts 
and crafts, sports, board games, fitness, 
and martial arts, but they should reflect 
student interests.^® To satisfy students’ 
nutritional needs, after-school programs 
should incorporate a snack time into the 
schedule, whereas summer programs 
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should consider including lunch or another 
appropriate meal/^ 



2. Promote awareness of the OST program 
within schools and to parents. 

It is the opinion of the panel that OST pro- 
grams, whether organized by schools, dis- 
tricts, or private providers, should consis- 
tently inform parents, teachers, and other 
school staff about programs and their ben- 
efits. Programs can use various methods 
of promotion, including websites, flyers 
distributed at parent meetings, notices on 
school bulletin boards or in school news- 
letters, and word of mouth. Information 
should include program location, hours 
of operation, and contact numbers so that 
parents can raise questions or concerns, 
and the information should be available in 
multiple languages when appropriate. 

Schools can join OST providers in promot- 
ing participation in the programs. For ex- 
ample, teachers and administrators can 
identify and recruit students who might 
benefit from OST program services. Teach- 
ers can provide referrals or informational 
materials during parent-teacher meetings or 
give the program a list of students in need 
of academic assistance.^^ Teachers or school 
administrators also can remind students at 
the end of the school day about attending 
the after-school program and, if needed, es- 
cort students directly to the program. 



3. Use attendance data to identify students 
facing difficulties in attending the program. 

Program coordinators should systemati- 
cally collect OST program attendance data 
and use the data to identify students with 
recurring absences or low attendance. OST 
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Staff can follow up with school staff to see 
if the problem extends to the school day. 
OST staff also could coordinate with school 
staff to contact parents to determine the 
reason for the absences.^® Programs can 
consider using reward incentives, positive 
reinforcement, or special privileges to en- 
courage regular attendance. For example, 
incentives can be in the form of monthly 
prizes or a point system that rewards stu- 
dents with points for good attendance or 
behavior that can be redeemed for special 
benefits such as field trips, school sup- 
plies, small prizes, or books. 

Potential roadblocks and solutions 

Roadblock 2.1. As students get older, ad- 
ditional options for after-school or summer 
recreational activities become available 
and increase the competition for students’ 
after-school timef^ 

Suggested Approach. An OST coordi 
nator should be aware of other extracur- 
ricular activities when making scheduling 
or timing decisions so that it is easier for 
students to participate in both academic 
and recreational programs and, thereby, 
minimize competition. OST programs also 
can make participation in sports practice 
or other activities a privilege contingent 
on attendance in the academic portion of 
the OST program. 

Roadblock 2.2. Students may have un- 
avoidable circumstances that prevent them 
from attending OST programs, such as tak- 
ing care of a sibling after school. 
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Suggested Approach. If a large portion 
of an OST program’s target group is fac- 
ing these types of barriers, the program 
should consider devising options for stu- 
dents who cannot be onsite. Programs 
might consider providing tutoring for 
these students in the home or implement- 
ing a system that can provide the instruc- 
tion in an online or electronic format. 

Roadblock 2.3. The number of slots in OST 
programs is limited. 

Suggested Approach. If the demand for 
OST programming is greater than the ca- 
pacity of the program, organizers should 
consider ways to expand their capacity 
or partner with other programs that have 
available space. Organizers may need to 
seek additional funding sources, such as 
foundations or other federal grant pro- 
grams, for which they might consider hir- 
ing an external evaluator to demonstrate 
the value of the program model. 



Roadblock 2.4. Communicating with 
families involved in the program may be 
difficult. 

Suggested Approach. Schools and OST 
providers should use multiple methods to 
communicate with parents. Some parents 
may move or change phone numbers fre- 
quently, which can make phone and mail 
communication difficult. Work with the 
school to provide information through 
school staff or teachers via flyers sent home 
with students. Recruitment efforts should 
extend beyond the school site to locations 
that families frequent. These may include 
grocery stores, laundromats, and com- 
munity and faith-based centers. Schools 
should consider the common languages 
spoken in the area and provide translated 
materials or hire bilingual staff. 
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Recommendatioii 3. 
Adapt instruction 
to individual and 
small group needs 

OST is an opportunity to supplement 
learning from the school day and to 
provide targeted assistance to students 
whose needs extend beyond what they 
can receive in the classroom. Since 
OST programs are shorter than the 
school day, instruction must be focused 
and targeted. The panel believes 
that closely aligning the content and 
pacing of instruction with student 
needs will result in better student 
performance. Determining the right 
level of difficulty and pace and the 
most appropriate skills to teach is 
critical to effectively individualizing 
instruction, but challenging in practice. 
To provide targeted help to a student, 
instructors need to assess and 
document students’ academic progress. 
Based on this assessment, students 
should be provided with instruction 
that accommodates their level of 
development and rate of learning.®' 

The same workbook or activity could 
be frustrating for some students 
and not challenging enough to be 
educational for others. If instructors 
are unfamiliar with ways to incorporate 
assessment or individualization into 
instructional time, they should be 
provided with the tools and support 
that will maximize their effectiveness. 

Level of evidence: Moderate 

The panel judged the level of evidence 
supporting this recommendation to be 
moderate. Learning environments that are 
adaptive to individual and small group 
needs are widely believed to be effective 
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in fostering achievement.®^ Within the con- 
text of OST, the literature is not definitive, 
and, in general, positive effects cannot be 
directly attributed to the use of the strat- 
egies outlined in this recommendation. 
However, looking more closely at the ac- 
tual implementation of practices related to 
this recommendation, there is a pattern of 
more positive academic effects associated 
with programs that more closely corre- 
spond to this recommendation. Therefore, 
the panel believes that OST programs can 
be more successful if they attempt to un- 
derstand the academic needs of the stu- 
dents they serve and adapt their programs 
to those needs. 

Brief summary of evidence to 
support the recommendation 

Of the 15 programs related to this recom- 
mendation with evaluations that met WWC 
standards with or without reservations,®® 
5 were judged to be in close correspon- 
dence with more than one aspect of this 
recommendation.®'' Four of these were 
found to have positive effects on academic 
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achievement,®^ and one had mixed but 
potentially encouraging effects.®® Of the 
remaining 10 programs,®^ 6 still showed 
positive or mixed effects on academics.®® 

Of the four programs with lower levels 
of relevance and without detectable ef- 
fects on academic achievement, two de- 
serve special mention because they are 
the two major sources of federal funding 
for academically focused OST programs: 
21st CCLC and SES, which are mandated 
as part of NCLB.®^ The national study of 
21st CCLC programs found no positive 
academic effects for either elementary 
or middle school students.^® However, 
implementation varied widely: although 
some programs reported tutoring in small 
groups with fewer than 10 students, most 
programs did not provide direct or adap- 
tive instruction that was geared to the 
individual needs of all their students. Al- 
though SES is a newer program relative 
to 21st CCLC and less often studied, it 
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appears to be implemented typically as 
one-on-one or small group tutoring (again 
implementation varies widely), and the re- 
sults from the states that have attempted 
to evaluate the effects of SES, as mandated 
by law, do not show significant impacts on 
state assessments.®^ 

In summary, the evidence demonstrates 
positive effects associated with a total of 
eight programs that adapted instruction 
to individual and small groups to some 
degree®^ and mixed effects in three other 
programs;®® however, because adapting 
instruction always was a component of a 
multicomponent intervention and because 
adapting instruction did not consistently 
demonstrate significant positive effects 
across every study reviewed, ®'^ the panel 
acknowledges that the level of evidence 
is moderate. 

How to carry out this 
recommendation 



1. Use formal and informal assessment data 
to inform academic instruction. 

OST programs should utilize the results of 
assessments administered to students dur- 
ing the school day — combined with input 
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from classroom teachers — to individual- 
ize instruction (see recommendation 1 for 
how to establish relationships with school 
staff). General assessment can measure a 
student’s content knowledge, appropriate 
difficulty level, mastery of a topic, or skills 
that require emphasis during instruction. 
The information gathered from assess- 
ments should be used to adapt the content, 
pace, and approach in instructing the stu- 
dent, whether in a one-on-one setting or 
in small groups.®^ 

If additional information about student 
progress is needed, OST instructors should 
incorporate formal and informal assess- 
ments into tutoring and homework assis- 
tance time.^® The types of tools instructors 
can use to assess students’ abilities and 
needs vary widely and should be deter- 
mined based on the OST program’s goals, 
the students involved, or other unique ex- 
periences. For example: 

• When students enter a program for the 
first time, a test can be administered to 
measure their baseline abilities. 

• During a lesson, instructors can 
use basic techniques such as effec- 
tive questioning^® and observation to 
gauge a student’s comfort with the ma- 
terial. For example, reading instructors 
can use a running record to evaluate 
reading behavior and keep track of 
mistakes that a student makes while 
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reading. The record can be used to 
identify trouble spots and monitor stu- 
dent progress over time. 

• Pre- and post-lesson exercises can pro- 
vide useful information on progress 
and can point the instructor to the 
areas in which additional support is 
required. 199 

These are just some examples of common 
assessment techniques; instructors should 
choose the tools they are comfortable with 
and those that gather the information they 
can use to adapt instruction in the most 
efficient way. 



2. Use one-on-one tutoring if possible; other- 
wise, break students into small groups. 

Ideally, OST programs should use one-on- 
one tutoring to provide academic instruc- 
tion to students. 191 The panel believes 
that a one-to-one ratio enables the most 
individualized attention for students and 
facilitates the continuous assessment of 
student progress and academic needs. i9^ 
If resources are limited and do not allow 
for one-on-one tutoring, the panel rec- 
ommends that students be broken into 
small groups of roughly three to nine 



99. In a running record, the instructor has a copy 
of the book the student is reading and follows 
along as the student reads. When the student 
makes a mistake, the instructor makes a note 
in his or her own copy. In subsequent readings 
of the same book, the instructor can identify 
which mistakes are repeated and which ones 
the student has learned to avoid (Iverson and 
Tunmer 1993). 

100. Black et al. (2008); Courtney et al. (2008). 

101. Morris, Shaw, and Perney (1990); Meier and 
Invernizzi (2001); McKinney (1995); Ross et al. 
(2008); Baker, Gersten, and Keating (2000). A 
meta-analysis of OST strategies found that one- 
on-one reading tutoring programs had larger 
average effect sizes than did reading programs 
that taught in small or large groups (Lauer et 
al. 2004). 

102. Slavin (2006). 



(26) 



3. ADAPT INSTRUCTION TO INDIVIDUAL AND SMALL CROUP NEEDS 



students, at least when there is the op- 
portunity for students to work indepen- 
dently.^'’^ In addition to giving the students 
an opportunity to learn teamwork skills 
and enhance their relationships with other 
students (see recommendation 4), divid- 
ing the class into smaller groups allows 
students to work at their own pace.’°^ Stu- 
dents who are more advanced might not 
feel frustrated by having to wait for other 
students to catch up, and students who 
struggle with the material would not feel 
the pressure of holding up the class.’”® 
Students can be grouped based on their 
skill level for that topic (indicated by as- 
sessment data) or by grade.’”'' Instructors 
also can create heterogeneous groups, in 
which students assist their peers in a co- 
operative learning format.’”® 



3. Provide professional development and ongo- 
ing instructional support to all instructors. 

The organizer of the OST program (i.e., 
the OST coordinator or the program di- 
rector) should have primary responsibility 



103. Black et al. (2008), Langberg et al. (2006), 
and U.S. Department of Education (2003) provide 
examples of small group sizes. 

104. Although the literature does not suggest 
conclusively that there are large differences in 
outcomes between one-on-one tutoring and small 
group instruction in out-of-school time, there 
is more evidence for the effectiveness of one- 
on-one tutoring in raising student achievement 
(Slavin 2006; Lauer et al. 2004; Bloom 1984). 

105. Schacter and Jo (2005); Black et al. (2008); 
Langberg et al. (2006); Udell (2003). 

106. In a survey of students in the Chicago Sum- 
mer Bridge program (Stone et al. 2005), research- 
ers reported that more advanced students were 
frustrated by the slow pace of the whole class 
and did not think that they were learning what 
they needed. Other students appreciated that the 
instructor would wait for them to understand 
the material before moving the class on to an- 
other topic. 

107. Schacter and Jo (2005); Black et al. (2008); 
Langberg et al. (2006); U.S. Department of Educa- 
tion (2003); Brown (2002). 

108. August et al. (2001); Johnson and Johnson 
(1999). 



for training instructors to implement the 
program properly. However, schools and 
districts already may have high-quality 
training and professional development 
resources. In this case, it may be to the 
school’s benefit to be involved in the train- 
ing of OST instructors, many of whom may 
be their own classroom teachers or para- 
professional staff, to ensure quality in- 
struction and promote alignment with the 
school curriculum. The panel recommends 
that schools discuss training options with 
the OST program and consider involving 
OST program instructors in training and 
professional development courses at the 
school and district levels. 

The level and intensity of OST instructor 
training should not overburden instruc- 
tors with unnecessary training but should 
ensure fidelity to the program. Further, the 
intensity of OST instructor training will de- 
pend on the background and experience 
of the instructors, including whether they 
are experienced and credentialed teach- 
ers, graduate students in education, or 
volunteers with minimal teaching expe- 
rience.’”” To facilitate targeted training, 
OST programs should vary the intensity of 
training by dividing teachers into separate 
tracks based on their prior experience.”” 
Inexperienced instructors should be ob- 
served and coached in the initial stages of 
teaching to monitor quality and to identify 
the need for additional training.’” OST 
programs can use experienced teachers as 
trainers and to serve as resources for less 
experienced instructors.”^ Training ses- 
sions should at least cover using assess- 
ment data in instruction, individualizing 
instruction and lesson planning, align- 
ing programming with the goals of the 



109. See recommendation 1 for a discussion of 
hiring staff for OST programs. 

110. Arbreton et al. (2008). 

111. Morris, Shaw, and Perney (1990); Udell 
(2003); Sheldon and Hopkins (2008); Wasik and 
Slavin (1993); Tucker (1995). 

112. Udell (2003). 
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school, and using monitoring and evalua- 
tion procedures. 

OST instructors should be given manuals 
to be used as an ongoing and useful refer- 
ence. All instructors also should receive 
ongoing support and professional develop- 
ment that is tailored to the needs of the 
instructors and their students or that tar- 
gets areas in which instructors are weak 
or need additional guidance. 

Potential roadblocks and solutions 

Roadblock 3.1. The program cannot af- 
ford one-on-one or small group tutoring. 

Suggested Approach. If not enough in- 
structors are available to give individual 
attention to each student, consider incor- 
porating volunteers or pooling instructors 
with other OST providers. Volunteers can 
help lower the student-staff ratio and can 
be used strategically to maximize the use 
of experienced teachers. High school stu- 
dents or other volunteers can be recruited 
to lead groups of students in recreational or 
athletic activities while experienced instruc- 
tors teach smaller groups or tutor individual 



113. Black etal. (2008); Udell (2003); Chaplin and 
Capizzano (2006); Arbreton et al. (2008); Court- 
ney et al. (2008); Wasik and Slavin (1993). 

1 14. August et al. (2001); Black et al. (2008); Lang- 
berg et al. (2006); Udell (2003); Chaplin and Capiz- 
zano (2006); Courtney et al. (2008). 

115. August et al. (2001); Black et al. (2008); Shel- 
don and Hopkins (2008). 



students. The program also could consider 
partnering with another provider that has 
sufficient staff resources and alternate staff 
schedules or combine programs. 

Roadblock 3.2. OST instructors are not fa- 
miliar with their students because they do 
not teach them during the school day. 

Suggested Approach. OST instructors 
have less time to get to know their students, 
especially academically, but the students’ 
classroom teachers have much of the infor- 
mation they may need. Instructors should 
take advantage of relationships with the 
school (see recommendation 1) to commu- 
nicate with the classroom teachers about 
students’ academic and social needs. OST 
instructors also may have more opportu- 
nities to communicate with parents when 
they pick up their children and should take 
advantage of that time to become more fa- 
miliar with their students. 

Roadblock 3.3. Useful assessment data 
are not currently available to individualize 
instruction. 

Suggested Approach. If the student as- 
sessment data available are not adequate 
for evaluating the needs of the student, 
the OST program should administer its 
own formal and/or informal assessments. 
The program or instructor should choose 
assessments that will provide the most 
information on the students’ achievement 
level with the least burden on the students 
or instructors. 
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Recommendation 4. 
Provide engaging 
learning experiences 

The panel recommends that OST 
activities be interactive, hands on, 
learner directed, and related to the 
real world, while remaining grounded 
in academic learning goals."® High- 
quality instruction is important, but 
producing achievement gains in OST is 
particularly challenging. Both students 
and teachers suffer from fatigue after a 
long school day or year. OST programs 
are typically voluntary and must 
compete with nonacademically oriented 
activities to attract students and effect 
learning gains. The panel believes 
that instructors must be particularly 
engaging to overcome student fatigue 
and distractions from nonacademically 
oriented activities. Although all 
of the practices outlined in this 
recommendation may not be relevant 
to all programs, the panel believes that 
OST programs have unique flexibility 
to provide engaging opportunities 
for student learning and that these 
recommendations can be useful in both 
one-on-one and group settings. 



Student engagement in school and 
classroom instruction is correlated 
with improved academic outcomes,"^ 
and disengagement is correlated 
with poor academic performance."® 
Student choice, cooperative learning 
experiences, and hands-on and real- 
world activities, as well as supportive 



116. Capizzano et al. (2007); Arbreton et al. 
(2008); Borman and Dowling (2006). 

117. Connell, Spencer, and Aber (1994); Marks 
(2000); Wellborn and Connell (1990); Connell and 
Wellborn (1991). 

118. Finn, Pannozzo, and Voelkl (1995); Finn and 
Rock (1997). 



relationships between staff and 
students, have been linked to student 
engagement, persistence with learning 
activities, and connection to the 
school."^ Evidence suggests that 
many of the activities discussed in 
this recommendation (e.g., games, 
recreation, or field trips) are ineffective 
when they occur independently of 
the academic component of the 
program.'^® However, the panel 
believes that by making the connection 
between engaging activities and 
academic learning explicit, OST 
programs can produce greater 
academic achievement gains. 

Level of evidence: Low 

The panel judged the level of evidence for 
this recommendation to be low. Studies 
of the types of activities covered in this 
recommendation have demonstrated that 
they are effective in laboratory and school- 
day settings. However, the evidence for 
whether these practices are effective is 
mixed in the OST context. Although many 
programs identified making programming 
engaging as a key program goal, very few 
demonstrated consistently positive ef- 
fects, and none linked positive effects di- 
rectly to the use of the strategies outlined 
in this recommendation. 



119. See, for example, Newmann (1991); Helme 
and Clark (2001); Blumenfeld and Meece (1988); 
Battistich et al. (1997); Klem and Connell (2004); 
Turner (1995); Perry (1998); Skinner and Belmont 
(1993); Connell and Wellborn (1991). This conclu- 
sion is drawn from the information in Fredricks, 
Blumenfeld, and Paris (2004), which reviews the 
available research on engagement. Other rele- 
vant sources include Cordova and Lepper (1996); 
Guthrie et al. (1999); and Guthrie, Wigfield, and 
VonSecker (2000). Students in Chicago Summer 
Bridge reported having more motivation when 
they perceived that their teachers were con- 
cerned with their well-being (Stone et al. 2005). 

120. U.S. Department of Education (2003). 
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Brief summary of evidence to 
support the recommendation 

Several studies, conducted outside of the 
OST arena, have examined the effectiveness 
of different strategies to increase student 
motivation, engagement, and academic suc- 
cess. In the OST context, however, the 
evidence supporting the use of engaging 
activities has been mixed. Five programs 
documented practices highly aligned with 
those recommended by the panel; these 
either made a deliberate effort to integrate 
engaging practices with academic con- 
tent or intentionally developed relation- 
ships between students and OST staff with 
the objective of engaging students with 
school and learning. Of these, three 
programs demonstrated positive academic 
effects, and two showed mixed effects. 
Six other programs had practices similar 
to the panel’s recommendations, but they 
either did not provide enough descriptive 
evidence to determine whether they were 
highly aligned with the panel’s recommen- 
dations or contained some evidence that 
the strategies were used in a way that was 
inconsistent with the panel’s recommen- 
dations. One of these showed positive 



121. Cordova and Lepper (1996); Anderson 
(1998); Guthrie et al. (1999); Guthrie, Wigfield, 
and VonSecker (2000); Battistich et al. (1997); 
Klem and Connell (2004); Helme and Clark (2001); 
Blumenfeld and Meece (1988); Connell and Well- 
born (1991). 

122. Early Risers — August et al. (2001); Develop- 
mental Mentoring — Karcher, Davis, and Powell 
(2002); Enhanced Academic Instruction — Black et 
al. (2008); Teach Baltimore — Borman and Dowl- 
ing (2006); KindergARTen — Borman, Goetz, and 
Dowling (2008). 

123. Early Risers — August et al. (2001); Devel- 
opmental Mentoring — Karcher, Davis, and Pow- 
ell (2002); KindergARTen — Borman, Goetz, and 
Dowling (2008). 

124. Enhanced Academic Instruction — Black et 
al. (2008); Teach Baltimore — Borman and Dowl- 
ing (2006). 

125. Summer Reading Day Camp — Schacter and 
Jo (2005); Chicago Summer Bridge — Jacob and 
Eefgren (2004); E.A.’s BEST— Goldschmidt, Huang, 
and Chinen (2007); YS-CARE — Bissell et al. (2002); 



effects, one showed mixed effects, and 
four showed no effects. Although the 
evidence is somewhat mixed, on average, 
those with positive or mixed effects seemed 
to be more closely aligned with the strate- 
gies recommended by the panel. 

How to carry out this 
recommendation 



1. Make learning relevant by incorporating 
practical examples and connecting instruc- 
tion to student interests and experiences. 

The panel recommends providing instruc- 
tion using tools or materials that students 
can relate to. OST staff should identify the 
academic concept being taught and then 
find practical examples and relevant ma- 
terial to support that learning objective. 
When possible, programs should consider 
integrating academic content using an over- 
arching program theme or final project to 
reinforce different learning activities and 
make learning more meaningful. Because 
OST programs operate during nontradi- 
tional hours (i.e., after school, weekends, 
or summers), they are positioned to pro- 
vide different types of activities than do 
schools. One example is field trips, which 
can help develop students’ background 
knowledge and connect the real world to 
the in-class curriculum. OST programs 
also can invite guest speakers to demon- 



21st CCLC — U.S. Department of Education (2003); 
NDP— Udell (2003). 

126. Summer Reading Day Camp — Schacter and 
Jo (2005). 

127. Chicago Summer Bridge — Jacob and Eef- 
gren (2004). 

128. E.A.'s BEST — Goldschmidt, Huang, and 
Chinen (2007); YS-CARE— Bissell et al. (2002); 
21st CCLC — U.S. Department of Education (2003); 
NDP— Udell (2003). 

129. Center for Applied Linguistics (1994); Re- 
isner et al. (2004); Karcher, Davis, and Powell 
(2002); Caplzzano et al. (2007); Borman, Goetz, 
and Dowling (2008). 

130. Borman and Dowling (2006); Borman, Goetz, 
and Dowling (2008). 
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strate how academic content relates to their 
career experiences to help students find 
practical meaning in the academic concepts 
they learn in school. Working or retired 
practitioners may be interested in serving 
as guest speakers and/or mentors. 

The panel believes that OST instructional 
strategies should capitalize on student in- 
terests and make students want to engage 
in the instructional material. OST staff 
should first develop a clear understanding 
of students’ interests. Brief conversations 
with students and/or teachers or quick sur- 
veys are simple mechanisms that programs 
can use to gather information about stu- 
dent interests.^^^ Instruction can then build 
off existing student interests and incorpo- 
rate examples from sports, current events, 
or other community-specific interests. OST 
instructors can connect reading materials 
or concepts introduced in class to students’ 
everyday life experiences. OST programs 
also can personalize instructional content 
and materials to student interests and pro- 
vide students with choices to maximize 
student learning. Instructional strate- 
gies can range from the simple (e.g., per- 
sonalizing reading instruction by begin- 
ning with the letters of a student’s name 
to teach letter-sound identification) to the 
elaborate (e.g., modifying a course text to 
make it relevant to students’ reading levels, 
experiences, and social contexts). 



2. Make learning active through opportuni- 
ties for collaborative learning and hands-on 
academic activities. 

The panel recommends that OST instruc- 
tion encourage students to think actively 



131. Capizzano et al. (2007); Chaplin and Capiz- 
zano (2006). 

132. Ferreira (2001); Center for Applied Linguis- 
tics (1994). 

133. Cordova and Lepper (1996). 

134. Schacter and Jo (2005); Black et al. (2008); 
Guthrie et al. (2000). 

135. Cordova and Lepper (1996). 



about and interact with academic content. 
One example is collaborative learning: 
OST programs can encourage interaction 
among peers by pairing struggling stu- 
dents with more advanced partners to 
help them grasp difficult concepts. OST 
instructors also can break students into 
groups to work together to solve a prob- 
lem or to rotate through learning stations, 
but effective group exercises can be less 
formal and as simple as having a group of 
three students complete a math problem 
together.13^ OST programs also can use 
role-playing activities to make experiences 
real and meaningful for students. 

Hands-on activities also can be helpful in 
reinforcing academic content.^^^ OST staff, 
particularly those working with younger 
students, should provide opportunities 
for students to use “exploration, creativ- 
ity, discovery and play.”^^° Games, projects, 
manipulatives, and computers can pro- 
vide practice and enrichment on content 
objectives. Hands-on science and math 
projects or exploratory learning activities 
make academic subjects interesting for 
students. Learning can be active without 
involving physical activity or elaborate les- 



136. Stone et al. (2005). 

137. Roberts and Nowakowski (2004). Ames 
(1992) describes the literature on how the learn- 
ing environment affects student motivation. 

138. Durlak and Weissberg (2007); Karcher(2005); 
James-Burdumy, Dynarski, and Deke (2007). 

139. Anderson (1998) 

140. Schacter and Jo (2005), p. 160. 

141. Black et al. (2008); Cordova and Lepper 
(1996); Capizzano et al. (2007); Nears (2008); 
Arbreton et al. (2008); Borman and Dowling 
(2006). 

142. Anderson (1998); Borman and Dowling 
(2006); Capizzano et al. (2007); Arbreton et al. 
(2008); Goldschmidt, Huang, and Chlnen (2007). 
Guthrie et al. (1999) found that the Concept- 
Oriented Reading Instruction (CORI) teaching 
framework, which includes the use of real-world 
observation and hands-on activities to personal- 
ize reading experiences, increased student moti- 
vation to learn and improved text comprehension 
and conceptual learning. 
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sons, however. OST programs should not 
underestimate the engaging nature of a 
dynamic instructor who uses active ques- 
tioning and participation to motivate and 
engage students in direct instruction. 



3. Build adult-student relationships among 
OST program participants. 

Positive and supportive relationships with 
adults can help students feel connected to 
the OST program and invested in the aca- 
demic material they cover. Likewise, as 
instructors get to know their students bet- 
ter, they can pinpoint their interests, relate 
academic content to their context and in- 
terests, and encourage them to have high 
expectations for their achievement. To 
do this, OST programs can hire staff with 
backgrounds and interests that comple- 
ment those of their students and can serve 
as positive role models and, thus, motivate 
students toward success. OST programs 
can use relationship-building activities to 
help staff get to know students and be- 
come invested in their outcomes. Pro- 
grams can support relationships between 
students and staff members by assigning 
one staff member to a group of students 
to move with them across OST activities 
or be with them as much as possible. As 
staff members spend more time with stu- 
dents, the panel believes they should en- 
deavor to accumulate knowledge about 
student interests and invest in supportive 
relationships,^^® helping students feel cared 
for and connected to both the OST program 
and academic learning more generally. 



143. Stone et al. (2005). 

144. Klem and Connell (2004); August et al. 
(2001); Karcher, Davis, and Powell (2002); Stone 
et al. (2005); Goldschmidt, Huang, and Chinen 
(2007); Udell (2003). 

145. Udell (2003). 

146. Center for Applied Linguistics (1994); Carter, 
Straits, and Hall (2007). 

147. Goldschmidt, Huang, and Chinen (2007); 
Udell (2003). 

148. Arbreton et al. (2008). 



OST programs also can support the devel- 
opment of other meaningful relationships 
in students’ lives. For example, mentors 
can help support relationships between 
students and their parents or teachers by 
coordinating and mediating meetings to 
discuss developmental needs or by cre- 
ating settings such as field trips in which 
mentees have the opportunity to interact 
socially with adults. 

Potential roadblocks and solutions 

Roadblock 4.1. The OST staff does not 
have experience leading cooperative learn- 
ing activities (or any of the other strategies 
outlined above). 

Suggested Approach. Unless teachers 
have experience using cooperative learn- 
ing, implementing these types of activi- 
ties can be difficult or ineffective. The 
panel recommends that OST programs 
hire teachers with these skills. OST pro- 
grams also can gather best practices from 
teachers at other sites and during the 
school day. Finally, OST programs should 
train staff to use these instructional strat- 
egies if they do not already have the ap- 
propriate skills. 

Roadblock 4.2. The academic content is 
too specific for one instructor to teach dif- 
ferent subjects to the same students. 

Suggested Approach. Particularly as stu- 
dents get older, it might not be feasible 
for the same staff member to instruct the 
same students across multiple subjects. 
Another method for developing positive 
relationships between students and OST 
instructors is establishing a “home-room” 
instructor whom students can engage with 
informally at a particular time during the 
program to develop a more meaningful 
and caring relationship. 



149. Karcher, Davis, and Powell (2002); Karcher 
(2005). 
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Roadblock 4.3. It’s too expensive to hire 
qualified staff to facilitate engaging aca- 
demic activities. 

Suggested Approach. To reduce costs, 
instead of hiring additional staff for the 
engaging activities designed to supple- 
ment academic content, OST programs can 
capitalize on existing staff with special- 
ized knowledge and experiences, use 
volunteers, or share enrichment staff with 
other OST programs. 



150. Borman and Dowling (2006). 

151. Bissell et al. (2002); Arbreton et al. (2008). 



Roadblock 4.4. Vke don’t have the re- 
sources to pay for field trips and materials 
for hands-on activities. 

Suggested Approach. Field trips and 
activities do not have to be elaborate. If 
resources are not available for field trips, 
walks outdoors to explore the surround- 
ing neighborhoods or parks or virtual field 
trips can be viable options. 



152. For information on virtual field trips, see 
Manzo (2009). 
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Recommendatioii 5. 
Assess program 
performance and 
use the results to 
improve the quality 
of the program 

OST program organizers should be 
aware of two types of performance 
assessments: (1) formof/Ve evaluations 
determine how a program has 
progressed in realizing its model of 
implementation and which program 
aspects are working well, and 
(2) summative evaluations determine 
how effective a program has been 
in achieving its goals, which usually 
include the improvement of student 
outcomes. Both types of evaluations 
are important in the life cycle of a 
program and especially instrumental 
in any program improvement effort.' 



For efficient program management, the 
panel believes that it is important for OST 
providers to have internal mechanisms 
to monitor staff performance and collect 
data related to program implementation. 
However, the panel believes that 
schools (or districts'^'') have a unique 
responsibility to conduct independent 
evaluations of program implementation 
and its impacts on students. The findings 
from these evaluations can be used to 



153. A meta-analysis of summer programs sug- 
gests that programs that undergo monitoring 
of instruction and performance tend to produce 
larger effects than programs that do not (Cooper 
et al. 2000). 

154. Throughout this recommendation, we refer 
to the school as the driving force behind the 
evaluation. However, it also is likely that districts 
will play this role, possibly because a program 
has wider coverage than just one school. Districts 
also may support schools in carrying out certain 
functions such as data collection or analysis, but 
that is a decision to be made collaboratively. 



spot problems and develop potential 
solutions, identify conditions in which 
the program is most effective, or make 
comparisons with the performance of 
other programs. The findings can be 
especially valuable in making long- 
term decisions about which strategies 
and programs should be continued or 
replicated in other areas. 

Level of evidence: Low 

The panel judged the level of evidence 
supporting this recommendation to be low. 
Although the panel believes that monitor- 
ing and improving performance is impor- 
tant to ensure that the program is carrying 
out its intended objectives and adapting to 
changing needs and feedback, no direct 
evidence suggests that monitoring leads 
to increased academic achievement in OST 
programs. More research would be needed 
to isolate whether it is the monitoring itself 
or other components that lead to positive 
academic effects in those programs that 
do some form of monitoring. 

Brief summary of evidence to 
support the recommendation 

OST programs could consider using differ- 
ent kinds of assessments. The panel found 
seven programs relevant to this recommen- 
dation with studies that met WWC stan- 
dards with or without reservations.'^^ Of 
these seven, five included elements of fidel- 
ity monitoring in order to ensure that pro- 
gram implementation followed the program 
model, one used fidelity monitoring and 



155. Early Risers — August et al. (2001); CHP — 
Langberg et al. (2006); Teach Baltimore — Borman 
and Dowling (2006); Chicago Summer Bridge — 
Jacob and Lefgren (2004); Enhanced Academic 
Instruction — Black et al. (2008); E.A.’s BEST — 
Goldschmidt, Huang, and Chinen (2007); YS- 
CARE— Bissell et al. (2002). 

156. Early Risers — August et al. (2001); CHP — 
Langberg et al. (2006); Teach Baltimore — Borman 
and Dowling (2006); Chicago Summer Bridge — 
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ongoing external evaluations, and one 
program used surveys of key stakeholders 
to gauge satisfaction with the program. 

Of the six that used fidelity monitoring, all 
were considered to be doing so in a way 
that was largely consistent with the panel’s 
recommendation, and five had either posi- 
tivei59 or partially positive effects on aca- 
demic achievement. The program that 
also used external evaluators and the pro- 
gram with stakeholder surveys were both 
inconclusive with nonsignificant effects. 
The one program that used stakeholder 
surveys was considered to have a lower 
level of consistency with this recommenda- 
tion because it was unclear how it used the 
surveys to improve the program. 

How to carry out this 
recommendation 



1. Develop an evaluation plan. 

An evaluation plan should present the 
evaluation objectives and research ques- 
tions, as well as details for the data collec- 
tion and analysis processes. This includes 
a decision about which type of evaluation 
is appropriate; formative evaluations are 
more relevant for new programs, whereas 
summative evaluations should be under- 
taken with more established programs^®^ 
that have demonstrated successful and 
consistent implementation for at least 



Jacob and Lefgren (2004); Enhanced Academic 
Instruction — Black et al. (2008). 

157. L.A.'s BEST — Goldschmidt, Huang, and 
Chinen (2007). 

158. YS-CARE— Bissell et al. (2002). 

159. Early Risers — August et al. (2001); CHP — 
Langberg et al. (2006). 

160. Teach Baltimore — Borman and Dowling 
(2006); Chicago Summer Bridge — ^Jacob and Lef- 
gren (2004); Enhanced Academic Instruction — 
Black et al. (2008). 

161. L.A.’s BEST — Goldschmidt, Huang, and 
Chinen (2007); YS-CARE— Bissel et al. (2002). 

162. This assumes that outcome data are avail- 
able and the sample size is large enough to have 
adequate statistical power to measure impacts. 



three years. The plan should contain 
information regarding the outcomes that 
will be used in the evaluation, the data 
that will be collected to measure those 
outcomes, and how data will be gathered. 
It also should outline the timeline for car- 
rying out various components of the plan 
and describe how results will be dissemi- 
nated and used. Since the process plays a 
defining role in the evaluation, the panel 
recommends that the school involve all 
stakeholders in the development of the 
plan, including teachers, parents, OST pro- 
gram administrators, and OST staff. 



2. Collect program and student performance 
data. 

Program implementation data, student out- 
come data, and feedback from other stake- 
holders regarding satisfaction with the 
program should be gathered. Program 
activity should be monitored as closely as 
possible; the more detail available about 
implementation, the easier it will be to 
identify specific areas for improvement. 
The panel believes that the school OST 
coordinator is the natural person to have 
a lead role in program monitoring, as that 
person will have the best understanding of 
both the school’s goals and the program’s 
operation. Some suggestions for collecting 
monitoring and outcome data follow: 

• Observations of implementation. Schools 
should observe OST instruction and 
student management, recreational 
time, and the day-to-day operation 
of the program (e.g., transportation, 
parent interaction, collaboration with 



163. Fullan (2001) suggests that school reform 
interventions should not be expected to demon- 
strate significant positive change in the first two 
to three years of implementation. 

164. A report by Ross, Potter, and Harmon (2006) 
provides useful guidance for states in evaluating 
SES providers, much of which can be applied to 
schools evaluating OST programs or providers. 

165. Black et al. (2008); Stone et al. (2005); Lang- 
berg et al. (2006); Borman and Dowling (2006). 
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schoolteachers). Create standardized 
tools for observers to use in assess- 
ing program quality.^®® In designing 
or choosing an observation tool, the 
school should decide which aspects of 
the program are important for effec- 
tive implementation, such as curricu- 
lum content, instructional delivery, and 
staff-student interaction. Each com- 
ponent should have its own checklist 
of items that breaks down the actions 
that represent good practice. Pro- 
viders also should conduct their own 
ongoing monitoring to ensure proper 
implementation. 

• Student outcomes. The panel recom- 
mends that the school and OST pro- 
vider share responsibility for collecting 
student outcome data. The OST pro- 
vider can track student attendance in 
the program, student performance on 
exercises or assessments conducted in 
the program, and student behavior such 
as engagement during instructional time 
or disciplinary incidences. The school 
will have access to course grades and 
records from the school day, as well as 
scores on state or district assessments. 
Recommendation 1 emphasized the pro- 
gram coordinator’s role as a facilitator 
in communicating between the school 
and the OST program regarding student 
academic needs and performance data. 
If needed, this role can be extended at 
the school level to recording data for 
evaluation purposes, including main- 
taining a central database of informa- 
tion collected from the OST program, 
district, and school staff regarding each 
student’s progress. 



166. Sheldon and Hopkins (2008). For more re- 
sources on program quality assessment tools, 
see Yohalem et al. (2009) and publications by the 
Bureau of Public School Options, Florida Depart- 
ment of Education (2007, n.d.). 

167. August et al. (2001). 

168. Bureau of Public School Options (2007). 

169. August et al. (2001); Borman and Dowling 
(2006). 



• Stakeholder satisfaction. The evalua- 
tion should provide an opportunity 
for stakeholders such as the principal, 
classroom teachers, parents, and stu- 
dents to offer input on how the pro- 
gram is meeting their needs, such as 
whether the OST curriculum is comple- 
menting classroom teaching or whether 
students are getting enough individual 
attention from instructors. At the 
basic level, the school could contact 
parents by phone to have an informal 
conversation about their child’s expe- 
rience with the OST program. Another 
option is to invite a small group of par- 
ents or teachers for a focus group dis- 
cussion about program performance. 
Mailing surveys is another alternative, 
since this option makes it possible to 
reach many respondents in a limited 
time frame. However, the panel cau- 
tions that even the most complex sur- 
veys have limitations in terms of bi- 
ased results and should be interpreted 
carefully. For example, surveys can 
be useful for identifying problematic 
trends in service delivery or areas for 
improvement, but they should not be 
interpreted as conclusive evidence of 
program effectiveness. 



3. Analyze the data and use findings for pro- 
gram improvement. 

The panel recommends that the school 
carefully analyze the data it has collected 
on implementation, student outcomes, 
and satisfaction and place findings in the 
context of how to improve the program. 
The school should look for inconsistencies 
between what OST providers proposed 
to do and how the program is actually 
implemented. They also should identify 
patterns in the data that suggest problem 
areas, such as irregular attendance on cer- 
tain days of the week. Districts can use a 



170. Goldschmidt, Huang, and Chinen (2007); 
Bissell et al. (2002). 
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larger dataset to look for patterns across 
schools or across OST providers. 

To encourage program growth, the panel 
recommends that the school share its 
evaluation results with the OST program. 
This also will allow suggestions for im- 
provement or strategies to address areas 
of concern to be discussed collaboratively. 
For example, if observations of instruction 
indicate that teachers are not adequately 
covering content, targeted professional 
development may be necessary. A for- 
mative evaluation that indicates that the 
program is not improving might suggest 
moving in a different direction from the 
current program model or even ending 
the program. 



4. Conduct a summative evaluation. 

Once formative evaluations conclude that 
the program is being implemented as de- 
signed, a summative or impact evaluation 
is appropriate. If three years or more have 
elapsed, a summative evaluation might 
still be useful even without consistent im- 
plementation. The summative evaluation 
should be conducted in the most rigorous 
way possible given the resources available 
to the school (or district) and the level of 
cooperation from the OST provider. A rigor- 
ous evaluation is one of the best measure- 
ments of a program’s success at raising 
student achievement, and positive results 
could boost participation in and funding 
for the program. An external evaluator 
can lend credibility to the evaluation and 
its findings. A randomized experiment 
is ideal but not always practical for every 
OST program. The school should consider 
the next best evaluation design that will 
provide evidence of effectiveness. For 



171. Goldschmidt, Huang, and Chinen (2007). 

172. See the What Works Clearinghouse (WWC) 
Procedures and Standards Handbook for a 
guide to quality research methods: http://ies. 
ed.gov/ncee/wwc/references/idocviewer/doc. 
aspx?docid=19&tocid=l. 



some larger districts, it might be feasible to 
conduct a district-wide, multischool evalu- 
ation, but for others the panel encourages 
participation in site-specific, independent 
evaluations. If a qualified evaluator is not 
available, the school should be cautious 
about concluding program effectiveness 
from the data collected for the evaluation. 
It could at least consider consulting with 
an experienced evaluator about evaluation 
plans and methods. 

Potential roadblocks and solutions 

Roadblock 5.1. Staff time and expertise 
are not sufficient for collecting and ana- 
lyzing data. 

Suggested Approach. If staff time and 
resources are already too constrained 
to apply to evaluation efforts, explore 
the availability of pro bono help from 
local colleges or universities or the par- 
ent-teacher association. These institutions 
and organizations may be able to provide 
observers or data collection and analysis 
services. Doctoral students might even be 
interested in conducting a full evaluation 
for their dissertation. 

Roadblock 5.2. The school and the pro- 
gram do not want to duplicate monitoring 
efforts. 

Suggested Approach. Both schools and 
OST providers need to be simultaneously 
collecting information on the program. 
The type of information each collects does 
not have be the same, and any overlapping 
responsibilities can be discussed in the 
evaluation planning stage. For example, 
providers may monitor attendance and 
collect data from their own instructors on 
assessments administered to students in 
OST, but schools will want to collect their 
own achievement data on the same stu- 
dents, including state assessments from 
the district. Both schools and OST provid- 
ers should conduct their own observations 
of instruction in the program. 
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Postscript from 
the Institute of 
Education Sciences 

What is a practice guide? 

The health care professions have embraced 
a mechanism for assembling and commu- 
nicating evidence-based advice to practitio- 
ners about care for specific clinical condi- 
tions. Variously called practice guidelines, 
treatment protocols, critical pathways, best 
practice guides, or simply practice guides, 
these documents are systematically devel- 
oped recommendations about the course of 
care for frequently encountered problems, 
ranging from physical conditions, such as 
foot ulcers, to psychosocial conditions, 
such as adolescent development. 

Practice guides are similar to the products 
of typical expert consensus panels in re- 
flecting the views of those serving on the 
panel and the social decisions that come 
into play as the positions of individual 
panel members are forged into statements 
that all panel members are willing to en- 
dorse. Practice guides, however, are gen- 
erated under three constraints that do not 
typically apply to consensus panels. The 
first is that a practice guide consists of a 
list of discrete recommendations that are 
actionable. The second is that those recom- 
mendations taken together are intended to 
be a coherent approach to a multifaceted 
problem. The third, which is most impor- 
tant, is that each recommendation is ex- 
plicitly connected to the level of evidence 
supporting it, with the level represented by 
a grade (e.g., strong, moderate, low). 

The levels of evidence, or grades, are usu- 
ally constructed around the value of par- 
ticular types of studies for drawing causal 
conclusions about what works. Thus, one 
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typically finds that a strong level of evi- 
dence is drawn from a body of random- 
ized controlled trials, the moderate level 
from well-designed studies that do not 
involve randomization, and the low level 
from the opinions of respected authorities 
(see Table 1). Levels of evidence also can be 
constructed around the value of particular 
types of studies for other goals, such as the 
reliability and validity of assessments. 

Practice guides also can be distinguished 
from systematic reviews or meta-analy- 
ses such as WWC intervention reviews or 
statistical meta-analyses, which employ 
statistical methods to summarize the re- 
sults of studies obtained from a rule-based 
search of the literature. Authors of prac- 
tice guides seldom conduct the types of 
systematic literature searches that are 
the backbone of a meta-analysis, although 
they take advantage of such work when 
it is already published. Instead, authors 
use their expertise to identify the most 
important research with respect to their 
recommendations, augmented by a search 
of recent publications to ensure that the 
research citations are up-to-date. Further- 
more, the characterization of the quality 
and direction of the evidence underlying a 
recommendation in a practice guide relies 
less on a tight set of rules and statistical al- 
gorithms and more on the judgment of the 
authors than would be the case in a high- 
quality meta-analysis. Another distinction 
is that a practice guide, because it aims for 
a comprehensive and coherent approach, 
operates with more numerous and more 
contextualized statements of what works 
than does a typical meta-analysis. 

Thus, practice guides sit somewhere be- 
tween consensus reports and meta-anal- 
yses in the degree to which systematic 
processes are used for locating relevant 
research and characterizing its meaning. 
Practice guides are more like consensus 
panel reports than meta-analyses in the 
breadth and complexity of the topic that 
is addressed. Practice guides are different 
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from both consensus reports and meta- 
analyses in providing advice at the level 
of specific action steps along a pathway 
that represents a more-or-less coherent 
and comprehensive approach to a multi- 
faceted problem. 

Practice guides in education at the 
Institute of Education Sciences 

lES publishes practice guides in education 
to bring the best available evidence and ex- 
pertise to bear on the types of challenges 
that cannot be addressed currently by sin- 
gle interventions or programs. Although 
lES has taken advantage of the history of 
practice guides in health care to provide 
models of how to proceed in education, 
education is different from health care in 
ways that may require that practice guides 
in education have somewhat different de- 
signs. Even within health care, for which 
practice guides now number in the thou- 
sands, there is no single template in use. 
Rather, one finds descriptions of general 
design features that permit substantial 
variation in the realization of practice 
guides across subspecialties and panels 
of experts. Accordingly, the templates 
for lES practice guides may vary across 
practice guides and change over time and 
with experience. 

The steps involved in producing an lES- 
sponsored practice guide are first to select 
a topic, which is informed by formal sur- 
veys of practitioners and requests. Next, a 
panel chair is recruited who has a national 
reputation and up-to-date expertise in the 
topic. Third, the chair, working in collabo- 
ration with lES, selects a small number of 
panelists to co-author the practice guide. 
These are people the chair believes can 
work well together and have the requi- 
site expertise to be a convincing source 
of recommendations. lES recommends 
that at one least one of the panelists be a 
practitioner with experience relevant to 
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the topic being addressed. The chair and 
the panelists are provided with a general 
template for a practice guide along the 
lines of the information provided in this 
appendix. They also are provided with 
examples of practice guides. The practice 
guide panel works under a short deadline 
of six to nine months to produce a draft 
document. The expert panel members in- 
teract with and receive feedback from staff 
at lES during the development of the prac- 
tice guide, but they understand that they 
are the authors and, thus, responsible for 
the final product. 

One unique feature of lES-sponsored prac- 
tice guides is that they are subjected to 
rigorous external peer review through 
the same office that is responsible for the 
independent review of other lES publica- 
tions. A critical task of the peer reviewers 
of a practice guide is to determine whether 
the evidence cited in support of particular 
recommendations is up-to-date and that 
studies of similar or better quality that 
point in a different direction have not been 
ignored. Peer reviewers also are asked to 
evaluate whether the evidence grade as- 
signed to particular recommendations by 
the practice guide authors is appropriate. 
A practice guide is revised as necessary to 
meet the concerns of external peer review- 
ers and to gain the approval of the stan- 
dards and review staff at lES. The process 
of external peer review is carried out inde- 
pendent of the office and staff within lES 
that instigated the practice guide. 

Because practice guides depend on the 
expertise of their authors and their group 
decisionmaking, the content of a practice 
guide is not and should not be viewed as a 
set of recommendations that in every case 
depends on and flows inevitably from sci- 
entific research. It is not only possible but 
also likely that two teams of recognized 
experts working independently to produce 
a practice guide on the same topic would 
generate products that differ in important 
respects. Thus, consumers of practice 
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guides need to understand that they are, 
in effect, getting the advice of consultants. 
These consultants should, on average, pro- 
vide substantially better advice than an 
individual school district might obtain on 
its own because the authors are national 



authorities who have to reach agreement 
among themselves, justify their recom- 
mendations in terms of supporting evi- 
dence, and undergo rigorous independent 
peer review of their product. 

Institute of Education Sciences 
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Megan Beckett, Ph.D., is a RAND sociolo- 
gist who has worked on a range of topics 
related to families and children, aging, 
and demography. Dr. Beckett is a recog- 
nized expert on after-school programs 
and policies and has co-authored a num- 
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speak on the subject of after-school care 
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assignment evaluation of the Quantum Op- 
portunity Program (QOP) demonstration, a 
multiservice program for high school-aged 
at-risk students, which included intensive 
mentoring and after-school academic sup- 
port. Dr. Schirm also directed the National 
Evaluation of Upward Bound, a program 
that provides extensive educational sup- 
port (including academic instruction, tutor- 
ing, and other services) for disadvantaged 



students preparing for college. He has 
served on four expert panels convened by 
the National Academy of Sciences’ Commit- 
tee on National Statistics and is a fellow of 
the American Statistical Association. 

Jessica Taylor, M.S., is program director 
at the Bureau of School Improvement at the 
Elorida Department of Education (EDGE), 
where she oversees a U.S. Department of 
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to their work as a panel. Panel members 
are asked to disclose their professional 
involvements and to institute deliberative 
processes that encourage critical exami- 
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they relate to the content of the practice 



guide. The potential influence of panel 
members’ professional engagements is 
further muted by the requirement that 
they ground their recommendations in 
evidence that is documented in the prac- 
tice guide. In addition, the practice guide 
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view prior to publication, with particular 
focus on whether the evidence related to 
the recommendations in the practice guide 
has been appropriately presented. 
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A search for research on out-of-school time 
(OST) programs in the United States from 
1988 to 2008 resulted in more than 1,000 
studies. Of these, 130 studies examined ac- 
ademically focused school-based OST pro- 
grams that serve elementary and middle 
school students and were reviewed accord- 
ing to What Works Clearinghouse (WWC) 
standards because they had a compari- 
son group and were deemed more likely 
to meet standards. Studies that included 
evidence of initial similarity between their 
treatment group and comparison group or 
attempted to account for the possible bias 
introduced by self-selection into the treat- 
ment group were considered more likely 
to meet standards. Twenty-two studies of 
18 programs met WWC evidence standards 
with or without reservations. 

Table D1 contains descriptive information 
about each of these programs, and Table 
D2 summarizes their relevance to each 
recommendation. Although several stud- 
ies contained practices that were similar to 
those recommended by the panel, in many 
cases, the studies did not describe them in 
sufficient detail to determine the degree to 
which the practices corresponded to those 
recommended by the panel. In other cases, 
there was evidence in the studies that the 
practices, although similar, were much less 
aligned with the panel’s recommendations 
or missed key parts. For example, several 
studies implemented engaging activities but 
failed to link them to an academic goal. 

It was challenging to determine the level 
of evidence for each recommendation be- 
cause programs’ actual practices coincide 
with the panel’s recommendations to vary- 
ing degrees, and many large OST programs 
have not demonstrated success in improv- 
ing academic outcomes. Studies of some 



OST programs found positive academic 
effects; other studies found no effects or 
mixed effects. Furthermore, OST programs 
necessarily contain multiple components 
that are related to parts of the recommen- 
dations in this guide, and the effects of a 
program cannot usually be causally attrib- 
uted to any particular component of that 
program. This creates a challenge for the 
guide, which aims to recommend specific 
practices rather than programs. To assess 
the importance of different components 
and their relevance to each recommen- 
dation, the panel reviewed implementa- 
tion reports of programs with evaluations 
that met WWC evidence standards (see 
the Introduction for a discussion of the 
WWC standards and their relevance to this 
guide). The level of evidence for each rec- 
ommendation was determined by consid- 
ering the number of programs that were 
related to each recommendation, the de- 
gree to which the programs implemented 
the recommendation, and the programs’ 
impacts on academic achievement. 

In almost all cases, the information cited 
about each program comes from a single 
study; however, there were two cases in 
which different studies measured the ef- 
fectiveness of a particular program (21st 
Century Community Learning Centers 
[21st CCLC] and supplemental educational 
services [SES]), and the panel relied on 
these to gather additional information 
on practices and impacts. In three other 
cases, more qualitative studies were con- 
sulted to gather additional descriptive 
evidence about a program with an impact 
study that met standards (Los Angeles’s 
Better Educated Students for Tomorrow 
[L.A.’s BEST], Chicago Summer Bridge, and 
Developmental Mentoring). Einally, when 
the panel judged it useful, recommen- 
dations were supplemented by evidence 
from outside the scope of OST or with 
evidence from studies that did not meet 
WWC standards. 
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Table D1. Studies of OST programs that met WWC standards with or without reservations 



Program and Study Details 


Brief Citation 


Program 

Type 


Grades 

Studied^ 


Program Length 


Sample 

Size 

(Analysis) 


Sample 

Characteristics 


Study 

Design 


Academic 

Assessment 

Measure 


Positive Academic Effects 


KindergARTen, 
Borman, Goetz, 

& Dowling (2008) 


Summer 


K 


6 hrs/day, 

5 days/week, 
6 weeks 


98‘> 


Urban, 
low-income, 
low-performing 
schools, largely 
non-White 


RCP 


DlBELS'i, word 
lists, DRA®, 
dictation 


Summer Reading 
Day Camp, Schacter 
&Jo (2005) 


Summer 


1 


9 hrs/day, 

5 days/week, 
7 weeks 


118 


Urban, 

low-income, 

largely 

non-White 


RCT 


Gates-MacGinitie 
and SAT^ 9 
Decoding and 
Comprehension 


Early Risers, 
August et al. (2001) 


Summer and 
school-year 
mentoring 


K-2 


4 days/week, 

6 weeks in 
summer; once a 
week during school 
year, 2 years 


201 


Semi-rural, 
low to low- 
middle income, 
Caucasian 


RCT 


Academic 

Competence® 


Developmental 
Mentoring, Karcher, 
Davis, & Powell 
(2002) 


Summer and 
school-year 
mentoring 


5 


1 Saturday/month, 
September-May; 

8 hrs/day, 

6 days/week for 
2 weeks in the 
summer 


26 


Low-performing 
school, largely 
non-White 


RCT 


WRAT-3h Spelling 


Experience Corps/ 
Book Buddies, 
Meier & Invernizzi 
(2001) 


After-school 

tutoring 


1 


45 min/session, 
~8 sessions/month, 
5-6 months 


56 


Urban, 
low-income, 
at-risk students 


RCT 


PALS* 


Start Making a 
Reader Today 
(SMART), Baker, 
Gersten, & Keating 
(2000) 


After-school 

tutoring 


1-2 


30 min/day, 
2 days/week, 
2 years 


84 


Low-performing 
schools, at-risk 
students, largely 
non-White 


RCT 


Woodcock Reading 
Mastery, Expressive 
One Word Picture 
Vocabulary, and 
three researcher- 
developed 
measures 


Howard Street 
Tutoring, Morris, 
Shaw, & Perney 
(1990) 


After-school 

tutoring 


2-3 


1 hr/day, 

2 days/week, 
October-May 


60 


Urban, 

low-income, 

low-achieving 


RCT 


Word Recognition, 
Spelling, and Basal 
Passage Reading 


Fast ForWord, 
Slattery (2003) 


After-school 

computer 

tutoring 


3-5 


100 min/day, 
5 days/week, 
~6 weeks 


60 


Urban, 

low-income 


RCT 


Yopp-Singer Test 
of Phoneme 
Segmentation, 
Qualitative Read- 
ing Inventory 11 


Challenging 
Horizons Program 
(CHP), Langberg 
etal. (2006) 


After school 


6-7 


2 hrs/day, 

4 days/week, 
one semester only 


48 


At-risk with 
behavior prob- 
lems, largely 
non-White 


RCT 


Teacher ratings of 
academic progress 


Mixed Academic Effects 


Teach Baltimore, 
Borman & Dowling 
(2006) 


Summer 


K-2 


6 hrs/day, 

5 days/week, 

7 weeks/summer 
for up to 3 summers 


686 


Urban, 

low-income, 

largely 

non-White 


RCT/ 

QEDJ 


CTBS'^ Reading 
Comprehension 
and Vocabulary 


Chicago Summer 
Bridge, Jacob 
& Lefgren (2004) 


Summer 


3, 6 


3 hrs/day, 
6 weeks 


-5000 


Urban, at-risk 
students, 
largely 
non-White 


RDD‘ 


ITBS™ Math 
and Reading 


Enhanced Academic 
Instruction, Black 
et al. (2008) 


After school 


2-5 


~3hrs/week, 
between 70 to 120 
days during 
first year 


1828 
Reading; 
1961 Math 


Low-income, 

largely 

non-White 


RCT 


SAT 10 Math 
and Reading" 



(continued) 
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Table D1. Studies of OST programs that met WWC standards with or without reservations (continued) 



Program and Study Details 


Brief Citation 


Program 

Type 


Grades 

Studied^ 


Program Length 


Sample 

Size 

(Analysis) 


Sample 

Characteristics 


Study 

Design 


Academic 

Assessment 

Measure 


No Detectable Academic Effects 


SES, McKay et al. 
(2008), Ross et al. 
(2008); Munoz, 
Potter, & Ross (2008) 


After-school 

tutoring 


K-12“ 


Varies by 
program 


Varies 
by state 


Low-performing 

schools 


QED 


State 

achievement 

tests 


Leap Frog, 
McKinney (1995) 


After-school 

tutoring 


1-2 


2 hrs/day, 

2 days/week, 
9 months 


44 Reading; 
47 Math 


Rural, 

low-achieving, 

largely 

non-White 


RCT 


SAT 8 Reading 
and Math 


Nurturing Develop- 
ment Partnerships 
(NDP), Udell (2003) 


After-school 

tutoring 


2 


~1 hr/d ay, 

2 days/week, 
10 weeks 


27 


Low-income, 
at-risk students, 
largely 
non-White 


QED 


Woodcock- 

Johnson-Revised 


L.A.’s BEST, 
Goldschmidt, 
Huang, & Chinen 
(2007) 


After school 


K-6 


5 days/week 


5662 


Urban, 
low-income, 
high ELL 


QED 


Normal Curve 
Equivalent scores 
of CTBSP, SAT O'!, 
and CAT 6' 


21st CCEC, U.S. 
Department of 
Education (2003); 
Dynarski et al. (2004); 
James-Burdumy 
etal. (2005) 


After school 


K-8 


2.5-3 hrs/day, 
4-6 days/week, 
9 months 


Elementary: 

1748; 

Middle: 

4068 


Low-performing 

schools, 

low-income 


RCT/ 

QED^^ 


Student records, 
SAT 9 


Youth Services — 
Child Care, 
Academic Assis- 
tance, Recreation, 
and Enrichment 
(YS — Care), Bissell 
et al. (2002) 


After school 


1-5 


4 hrs/day 


660 

Reading; 
672 Math 


Urban, 

low-income, 

low-achieving 


QED 


SAT 9 Reading 
and Math 



a. For summer programs, refers to the grade participating students exited prior to the commencement of the summer program. 

b. The sampie size in the KindergARTen evaluation varied by outcome. The size of the treatment group was either 72 or 73 students for four out of 
five of the literacy outcomes tested and was 60 students for the Development Reading Assessment (DRA). The size of the control group varied from 
23 to 27 students across the five outcomes. 

c. RCT stands for randomized controlled trial. 

d. Center on Teaching & Learning, (n.d.). DIBELS data system. Eugene, OR: University of Oregon. Retrieved May 29, 2009, from https://dibels.uoregon.edu/. 

e. Beaver, J., & Varter, M. (n.d.). Developmental reading assessment (2nd ed.). Lebanon, IN: Pearson. Retrieved May 29, 2009, from http://www.pearson- 
school.com/. 

f. Gates-MacGinitie Reading Tests. (2000). GMRT validity and reliability statistics. Itasca, IL: Riverside Publishing. 

g. Academic competence is a composite variable based on the Woodcock-Johnson-Revised and parent and teacher ratings. Woodcock, R. W., & Johnson, 
M. B. (1990). Woodcock-Johnson Psychoeducationai Battery — Revised: Tests of Achievement. Allen, TX: DML Teaching Resources. 

h. Jastak, S., & Wilkinson, G. (1994). Wide Range Achievement Test (3rd ed.). San Antonio, TX: The Psychological Corporation. 

i. Curry School of Education, (n.d.). Phonological Awareness Literacy Screening. Charlestown, VA: University of Virginia. Retrieved May 29, 2009, from 
http://pals.virginia.edu/. 

j. QED stands for quasi-experimental design. 

k. McGraw-Hill. (n.d.). Comprehensive Test of Basic Skills. Monterey, CA: MacMillan/McGraw-Hill. Retrieved May 29, 2009, from http://www.ctb.com/. 

l. RDD stands for regression discontinuity design, in this case, students who were right above the cutoff for mandatory summer school were com- 
pared to students who were right below the cutoff and, therefore, had to attend summer school. Since all tests have some margin of error in judging 
students' academic abilities, students near the cutoff could be very similar in actual abilities. Furthermore, the test score pre-summer school and 
other baseline characteristics are controlled for when looking at differences in students the next year. 

m. College of Education, (n.d.). Iowa Test of Basic Skills. Iowa City, lA: The University of Iowa. Retrieved May 29, 2009, from http://www.education.uiowa. 
edu/itp/itbs/. 

n. The Stanford Achievement Test, 10th ed., abbreviated battery for either math or reading was given to students in the beginning and end of school year. 
For 2nd and 3rd graders, the Dynamic Indicators of Basic Early Literacy Skills also was used. Harcourt Assessment, (n.d.). Stanford Achievement Test 
Series, Tenth Edition-Abbreviated Battery, Areas of Assessment. San Antonio, TX: Author. Retrieved September 9, 2007, from http://harcourtassessment. 
com/HAlWEB/Cultures/en-us/Productdetail.htm?Pid=SAT10Ac&Mode=summary&Leaf=SAT10A_2; Harcourt Assessment. (2003). Stanford Achievement 
Test Series, Tenth Edition-Spring Multilevel Norms Book. San Antonio, TX: Author.; Harcourt Assessment. (2004). Stanford Achievement Test Series, Tenth 
Edition-Technical Data Report. San Antonio, TX: Author. 

0 . Grades included in SES could be any from kindergarten to 12th grade. Typically, grades included in state or district evaluations are those that are 
more often tested statewide with tests that are comparable across grade levels, usually 3rd through 8th. 

p. McGraw-Hill. (n.d.). Comprehensive Test of Basic Skills. Monterey, CA: MacMillan/McGraw-Hill. Retrieved May 29, 2009, from http://www.ctb.com/. 

q. Pearson, (n.d.). The Stanford Achievement Test, 10th ed. San Antonio, TX: Author. Retrieved May 29, 2009, from http://pearsonassess.com. 

r. McGraw-Hill. (n.d.). California Achievement Test. Monterey, CA: MacMillan/McGraw-Hill. Retrieved May 29, 2009, from http://www.ctb.com/. 

s. Results for elementary students are from an RCT study, whereas the middle school study is a Q.ED. 
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Table D2. Studies and corresponding recommendations 



Program and Study Details 


Recommendations^ 


Brief Citation 


Program 

Type 


Grades 

Studied'’ 


1. 

Align 


2. 

Attendance 


3. 

Individualize 


4. 

Engage 


5. 

Assess 


Positive Academic Effects 


KindergARTen, 
Borman, Goetz, 

& Dowling (2008) 


Summer 


K 




V 


V 


VV 




Summer Reading Day 
Camp, Schacter & Jo 
(2005) 


Summer 


1 




V 


V 


V 




Early Risers, August 
et al. (2001) 


Summer and 
school-year 
mentoring 


K-2 


V 


V 


VV 


VV 


VV 


Developmental 
Mentoring, Karcher, 
Davis, & Powell (2002) 


Summer and 
school-year 
mentoring 


5 








VV 




Experience Corps/Book 
Buddies, Meier 
& Invernizzi (2001) 


After-school 

tutoring 


1 






V 






SMART, Baker, Gersten, 
& Keating (2000) 


After-school 

tutoring 


1-2 




V 


V 






Howard Street 
Tutoring, Morris, 
Shaw, & Perney (1990) 


After-school 

tutoring 


2-3 




V 


VV 






Fast ForWord, Slattery 
(2003) 


After-school 

computer 

tutoring 


3-5 




V 


VV 






CHP, Langberg et al. 
(2006) 


After school 


6-7 


w 


V 


VV 




VV 


Mixed Academic Effects 


Teach Baltimore, 

Borman & Dowling (2006) 


Summer 


K-2 


V 


V 


V 


VV 


VV 


Chicago Summer Bridge, 
Jacob & Lefgren (2004) 


Summer 


3, 6 


VV 




V 


V 


VV 


Enhanced Academic 
Instruction, Black et al. 
(2008) 


After school 


2-5 


V 


VV 


VV 


VV 


VV 


No Detectable Academic Effects 


SES, McKay et al. (2008); 
Ross et al. (2008); Munoz, 
Potter, & Ross (2008) 


After-school 

tutoring 


K-12‘: 


V 




V 






Leap Frog, McKinney 
(1995) 


After-school 

tutoring 


1-2 


VV 


V 


V 






NDP, Udell (2003) 


After-school 

tutoring 


2 


V 


V 


V 


V 




L.A.’s BEST, Goldschmidt, 
Huang, & Chinen (2007) 


After school 


K-6 


V 


V 




V 


VV 


21st CCLC, U.S. 
Department of Education 
(2003); Dynarski et al. 
(2004); James-Burdumy 
et al. (2005) 


After school 


K-8 


V 


V 


V 


V 




YS-CARE, Bissell et al. 
(2002) 


After school 


1-5 


V 


V 




V 


V 



a. Indicates relevance of recommendation to program operation. VV” indicates high relevance; V” indicates lower relevance. 

b. For summer programs, refers to the grade participating students exited prior to the commencement of the summer program. 

c. Grades included in SES could be any from kindergarten to 12th grade. Typically, grades included in state or district evaluations are those that are 
more often tested statewide with tests that are comparable across grade levels, usually 3rd through 8th. 



48 








APPENDIX D. TECHNICAL INFORMATION ON THE STUDIES 



In the remainder of this appendix, we 
summarize the evidence relevant to each 
of the panel’s recommendations. First, we 
briefly describe the level of evidence sup- 
porting each recommendation. Next, we 
provide a brief description of the char- 
acteristics and results of the studies that 
support each recommendation, the prac- 
tices described in those studies, and the 
degree to which they align with the panel’s 
recommendations. 

In general, each description first describes 
the relevant studies that produced posi- 
tive academic effects. Next, we review the 
studies that showed mixed effects (posi- 
tive results in only some subgroups or, in 
the case of one study that used a mixed 
design, positive results from the quasi-ex- 
perimental portion of the study but not the 
experimental portion). The studies that 
showed no positive effects on academic 
outcomes are described next, followed by 
supplemental evidence from studies that 
either were not eligible for WWC review or 
did not meet WWC standards. 

Recommendation 1. 

Align the OST program 
academically with the school day 

Level of evidence: Low 

The level of evidence for this recommen- 
dation is low. There is no direct evidence 
that practices outlined in this recommen- 
dation contribute to improved academic 
outcomes. Although it was common for 
programs to include some components of 
the panel’s recommendations, none tested 
the effectiveness of this recommendation 
individually, only in combination with the 
other components of OST programs. In the 
panel’s opinion, collaboration can improve 
academic outcomes and in the studies re- 
viewed for this guide, two independent 
evaluators recommended that collabora- 
tion between in-schooI time and OST be 



strengthened if possible. However, we 
fully acknowledge that more research is 
required to explore whether there is a con- 
nection between collaboration and posi- 
tive academic outcomes in practice. 

Summary of evidence 

Fifteen OST programs endeavored to 
collaborate with school-based staff or 
initiatives,^^® but, in general, these efforts 
were not core components of the programs, 
and three programs expressed difficulty or 
reluctance to coordinate more fully.^^^ Of 
the 11 programs with studies that met WWC 
standards with or without reservations,^^® 



175. Schacter and Jo (2005). Center for Applied 
Linguistics (1994) suggested the use of more 
collaboration when appropriate. 

176. CHP — Langberg et al. (2006); Early Risers — 
August et al. (2001); Enhanced Academic Instruc- 
tion — Black et al. (2008); Teach Baltimore — Borman 
and Dowling (2006); Chicago Summer Bridge— Ja- 
cob and Lefgren (2004); L.A.’s BEST— Goldschmidt, 
Huang, and Chinen (2007); YS-CARE — Bissell et al. 
(2002); 21st CCLC — U.S. Department of Education 
(2003); Leap Frog — McKinney (1995); NDP — Udell 
(2003); SES— McKay et al. (2008); SES— Ross et 
al. (2008); SES — Munoz, Potter, and Ross (2008); 
Title I supplementary education — Borman (1997); 
TASC — Reisner et al. (2004); Project Adelante — 
Center for Applied Linguistics (1994); After school 
tutoring — Leslie (1998). 

177. In James-Burdumy et al. (2005), the authors 
noted that 21st CCLC programs struggled to effec- 
tively coordinate homework help with the school; 
and in U.S. Department of Education (2003), 21st 
CCLC programs were found to be supportive but 
not “integrated" (p. 39) with the school. In Project 
Adelante (Center for Applied Linguistics 1994), 
program directors recommended closer coordina- 
tion with school-day staff to collect data and share 
information on student progress but were con- 
cerned about the appropriateness of the school- 
day curriculum for the students that their program 
served. Similarly, Morris, Shaw, and Perney (1990) 
expressed reluctance to align their after-school tu- 
toring to the school curriculum given that they ob- 
served that their students’ classroom instruction 
was often above their current reading levels. 

178. CHP — Langberg et al. (2006); Early Ris- 
ers — August et al. (2001); Enhanced Academic 
Instruction — Black et al. (2008); Teach Balti- 
more — Borman and Dowling (2006); Chicago 
Summer Bridge — ^Jacob and Lefgren (2004); L.A.’s 
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only 3 documented practices that closely 
corresponded to the panel’s recommenda- 
tions: CHP, Leap Frog, and Chicago Summer 
Bridge (see Table D3).^^^ In CHP and Leap 
Frog, coordination between schoolteach- 
ers and OST instructors was frequent and 
structured. Content and skills taught 
during OST were intentionally designed to 
support students during their school-day 
instruction. CHP showed positive effects 
and Leap Frog did not.^®^ The purpose of 
Chicago Summer Bridge was to help stu- 
dents achieve proficiency on state exami- 
nations that they had not mastered during 
the school year. Because it was a summer 
school program, coordination with individ- 
ual teachers was limited, but the curricu- 
lum was designed by the district with the 
express purpose of helping students meet 
state standards and closely linked to that 
goal. It had significant persistent effects on 
both math and reading for 3rd graders but 
not for 6th graders. 

The remaining eight programs included 
some components similar to the action 
steps for this recommendation, but the 
nine studies of the programs indicated that 
the degree of coordination was much less, 
or not enough information was provided 
to determine the level of alignment be- 
tween programs. Of these, one program 



BEST— Goldschmidt, Huang, and Chinen (2007); 
YS-CARE— Bissell et al. (2002); 21st CCLC— U.S. 
Department of Education (2003); Leap Erog — 
McKinney (1995); NDP— Udell (2003); SES— McKay 
et al. (2008); SES — Ross et al. (2008); SES — Munoz, 
Potter, and Ross (2008). 

179. CHP — Langberg et al. (2006); Chicago Sum- 
mer Bridge — Jacob and Lefgren (2004); Leap 
Prog — McKinney (1995). 

180. CHP — Langberg et al. (2006); Leap Frog — 
McKinney (1995). 

181. Ibid. 

182. Jacob and Lefgren (2004). 

183. Early Risers — August et al. (2001); Enhanced 
Academic Instruction — Black et al. (2008); Teach 
Baltimore — Borman and Dowling (2006); L.A.’s 
BEST— Goldschmidt, Huang, and Chinen (2007); 
YS-CARE— Bissell et al. (2002); 21st CCLC— U.S. 
Department of Education (2003); NDP — Udell 



showed positive effects, two showed 
mixed effects, and five showed no de- 
tectable academic effects.^®® Although more 
of these programs failed to demonstrate 
effectiveness. Table D2 shows that there 
were seven effective programs that did not 
attempt coordination with schools. Given 
the absence of a clear pattern of effective- 
ness based on the level of coordination with 
schools and the small sample of programs 
with high levels of coordination, the panel 
decided that the level of evidence was low. 



Positive evidence for alignment 
between in-school time and OST 

Two small to medium randomized con- 
trolled trials of after-school programs, CHP 
and Early Risers, attempted to coordinate 
with the schools that their students at- 
tended and showed positive effects, but 
only in CHP did its coordination practices 
seem to be both similar to the specific action 
steps recommended by the panel and an 
important part of the overall program.^®® 

Langberg et al. (2006) used a randomized 
controlled trial to study small group CHP, 
which provided after-school academic 
remediation and study skills training for 
6th and 7th graders with a combination of 
learning and behavior problems. The pro- 
gram showed no statistically significant 



(2003); SES— McKay et al. (2008); SES— Ross et al. 
(2008); SES — Munoz, Potter, and Ross (2008). 

184. Early Risers — August et al. (2001). 

185. Enhanced Academic Instruction — Black et 
al. (2008); Teach Baltimore — Borman and Dowl- 
ing (2006). 

186. L.A.’s BEST — Goldschmidt, Huang, and 
Chinen (2007); YS-CARE— Bissell et al. (2002); 
21st CCLC — U.S. Department of Education (2003); 
NDP— Udell (2003); SES— McKay et al. (2008); 
SES — Ross et al (2008); SES — Munoz, Potter, and 
Ross (2008). 

187. Langberg et al. (2006) had a sample size 
of 48 students; August et al. (2001) had 201 
students. 

188. Langberg et al. (2006). 
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Table D3. Studies of programs cited in recommendation 1 that met WWC standards 
with or without reservations 



Program and Study Details 


Strategies 


Brief Citation 


Program 

Type 


Grades 

Studied^ 


Level of 
Relevance*’ 


OST 

Coordinator 
Maintains 
Relations 
with School 


School Staff 
Communicate 
with OST 
Program 


OST 

Instruction Is 
Coordinated 
with School 
Goals 


School 
Helped 
Identify 
Staff for OST 
Programs 


Positive Academic Effects 


Early Risers, August 
et al. (2001) 


Summer 

and 

school-year 

mentoring 


K-2 


V 


To support 
classroom 
teachers 








CHP, Langberg et al. 
(2006) 


After 

school 


6-7 


VV 


Weekly 


Classroom 
teacher identi- 
fied student 
problem areas 


District 

determined 

academic 

content 




Mixed Academic Effects 


Teach Baltimore, 
Borman & Dowling 
(2006) 


Summer 


K-2 


V 






Reported 
alignment 
with school- 
year curricula 




Chicago Summer Bridge, 
Jacob & Lefgren (2004) 


Summer 


3, 6 


VV 






District 
designed 
curricula; 
intended to 
help students 
pass state tests 


Principals 
selected 
teachers 
for summer 
program 


Enhanced Academic 
Instruction, Black et al. 
(2008) 


After 

school 


2-5 


V 


Part-time OST 
coordinator 
at each site 






Majority of 
OST instruc- 
tors teach 
at school 


No Detectable Academic Effects 


SES, McKay et al. (2008); 
Ross et al. (2008); 
Munoz, Potter, & Ross 
(2008) 


After- 

school 

tutoring 


K-12'^ 


V 


Varies 


Varies 


Varies 


Varies 


Leap Frog, McKinney 
(1995) 


After- 

school 

tutoring 


1-2 


VV 


Weekly 
written 
comments to 
teachers 


Teachers 
sent student 
assignments 






NDP, Udell (2003) 


After- 

school 

tutoring 


2 


V 






Reported 
alignment 
with school- 
day learning 




L.A.’s BEST, 
Goldschmidt, Huang, 
& Chinen (2007) 


After 

school 


K-6 


V 


Informal 


Principal 

buy-in 

required before 
program starts 






21st CCLC, U.S. Depart- 
ment of Eciucation (2003); 
Dynarski et al. (2004); 
James-Burdumy et al. 
(2005) 


After 

school 


K-8 


V 


Varies 




Varies 


Varies 


YS-CARE, Bissell et al. 
(2002) 


After 

school 


1-5 


V 






Reported 
alignment 
with school- 
day curricula 





a. For summer programs, refers to the grade participating students exited prior to the commencement of the summer program. 

b. indicates relevance of recommendation to program operation. VV” indicates high relevance; V” indicates lower relevance. 

c. Grades included in SES could be any from kindergarten to 12th grade. Typically, grades included in state or district evaluations are those that are 
more often tested statewide with tests that are comparable across grade levels, usually 3rd through 8th. 
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effects on teacher ratings of academic 
progress because of the small sample 
size, but the effect size (calculated using 
Hedges’s was positive at 0.45d^o In 
order to acknowledge meaningful effects 
regardless of sample size, the panel fol- 
lowed WWC guidelines and considered a 
positive statistically significant effect or 
an effect size greater than 0.25 as an indi- 
cator of improved outcomes. 

CHP was implemented as an alternative 
to the district-run after-school program 
in Columbia, South Carolina. Sixth- and 
Zth-grade students were recruited based 
on poor performance on state exams. The 
first group was randomly assigned to par- 
ticipate in CHP the first semester, and the 
second group began participation in the 
second semester of the same year. The 
program staff undertook considerable ef- 
forts to maintain communication with, and 
support the services of, the school day to 
improve students’ behavior, organization, 
and academic achievement. Districts deter- 
mined the content of the program, which 
ensured that it was aligned with the dis- 
trict curriculum. Students in the program 
received extra privileges for correctly re- 
cording their homework assignments and 
keeping their school materials organized, 
encouraging the development of skills for 
school success. Students in various groups 
were required to demonstrate the use of 
learned skills at school, and interventions 
targeted problem areas identified by teach- 
ers. Each staff member in the OST program 
was assigned to maintain weekly contact 
with two teachers regarding upcoming as- 
signments and classroom behavior.^^^ 

A study of Early Risers, a two-year tutor- 
ing and mentoring program that recruits 
children with early onset aggressive be- 



189. See Appendix B of WWC Procedures and 
Standards Handbook. 

190. Langberg et al. (2006). 

191. Ibid. 

192. Ibid. 



havior at the end of kindergarten, showed 
positive academic effects but utilized dif- 
ferent methods of collaboration than those 
discussed in this recommendation. Pro- 
gram staff met regularly with classroom 
teachers to review student progress and 
discuss instructional practices, but they 
served more as a resource to teachers than 
vice versa. The study demonstrated sig- 
nificant increased academic competence 
for the treatment group relative to a com- 
parison group, but behavioral effects were 
not significant.^®^ 



Mixed evidence for alignment between 
in-school time and OST 

Three other larger studies of programs 
with varying degrees of school coordina- 
tion had mixed yet potentially encouraging 
findings. The first study with mixed ef- 
fects was a regression discontinuity study 
of Chicago Summer Bridge, a mandatory 
summer school program for students in 
grades 3 and 6 who failed to meet stan- 
dards for promotion on state tests. The 
study compared the achievement gains of 



193. August et al. (2001). 

194. Ibid. Academic competence is a compos- 
ite measure constructed by the authors using 
the Woodcock-Johnson Tests of Achievement- 
Revised (the broad reading and applied prob- 
lems domains) and several components of the 
teacher and parent ratings. The teacher ratings 
included in the composite were the Learning 
Problems scale from the Behavioral Assessment 
System for Children-Teacher Rating Scale, the 
Cognitive Competence scale from the Teacher’s 
Scale of Child’s Actual Competence and Social 
Acceptance, and the Concentration Problems 
scale from the Teacher Observation of Classroom 
Adaptation-Revised. The parent rating was the 
Concentration Problems scale from the Parent 
Observation of Classroom Adaptation used to 
assess behavior at home. 

195. Borman and Dowling (2006); Jacob and Lef- 
gren (2004); Black et al. (2008). 

196. Jacob and Lefgren (2004). The standards for 
promotion also apply to 8th graders in Chicago 
but only 3rd and 6th graders were included in 
the impact study. 
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students with June test scores just above 
and just below the cutoff for mandatory 
summer school in order to potentially be 
promoted to the next grade. It showed 
positive effects on student achievement 
in both reading and math among the dis- 
tricts’ lowest-performing 3rd-grade stu- 
dents, but not among 6th graders. In this 
program, coordination was fairly strong 
as the summer program was specifically 
designed to help students achieve the 
academic standards they failed to meet 
during the school year. Instructors used a 
centrally designed curriculum distributed 
by the district with lesson plans included, 
which emphasized the basic skills needed 
to pass the test at the end of the summer 
and be promoted.^®® Summer instructors 
were selected by principals. 

In two additional studies, alignment with 
schools was less strong or not described 
in the same detail. The second of the 
three studies with mixed effects was of 
Enhanced Academic Instruction in After- 
School Programs, which separately evalu- 
ated math and reading programs designed 
to improve the academic instruction in 
after-school instruction for children in 
grades 2 through 5.^°° Students were ran- 
domly assigned to receive the regular 
after-school-day offering (the comparison 
group) or either one of the structured en- 
hanced curricula (Harcourt School Publish- 
ers’ Mathletics at centers in the math study 



197. Students took the state exam again at the 
end of the summer program. If they failed to meet 
standards a second time, they would be retained 
in the same grade level. Between the spring of 
1997 and the spring of 1999, 43 percent of 3rd 
graders and 31 percent of 6th graders failed in 
either reading or math the first time in June. Of 
those who failed the first time, 48 percent of 3rd 
graders and 39 percent of 6th graders were ulti- 
mately retained (Jacob and Lefgren 2004). 

198. Jacob and Lefgren (2004); supplementary 
information from Stone et al. (2005); Roderick, 
Jacob, and Bryk (2004). 

199. Supplementary information from Roderick, 
Jacob, and Bryk (2004). 

200. Black etal. (2008). 



or Success for All Foundation’s Adventure 
Island program at the reading centers). 
Managing sites were provided with fund- 
ing for a part-time district coordinator who 
was familiar with the district and students 
served and who coordinated several man- 
agement tasks associated with the imple- 
mentation of the program. Three-quarters 
of the program instructors were class- 
room teachers from the school day in the 
school in which the program was located. 
In some cases, the program staff worked 
with classroom teachers and principals 
to identify eligible students. The curri- 
cula were tied to academic standards, but 
some participating instructors reported 
inconsistencies between the strategies and 
vocabulary used for instruction during the 
school day and those used during the en- 
hanced OST program. 

The enhanced program was characterized 
by increased instructional time and spe- 
cific strategies regarding staffing, support 
for staff, and student attendance efforts 
that distinguished it from the regular pro- 
gram. The evaluation found a significant 
impact on math scores with an effect size 
of 0.06 and no significant impacts on read- 
ing scores. Program impacts were mea- 
sured relative to students who participated 
in a regularly offered after-school program 
that typically consisted of homework help 
or unstructured direct instruction. This 
study is not a general evaluation of the im- 
pacts of an after-school program; instead, 
it is a test of a particular program type- 
structured curricula in math or reading 
versus more typical after-school program- 
ming. Although the available evidence sug- 
gests that coordination with schools was a 
component of both the reading and math 
programs, the study does not suggest 
that these were a major component of ei- 
ther program. In general, program effects 



201. Based on classroom teacher surveys, the 
program was not associated with increased in- 
cidents of behavior problems or reduced home- 
work completion during the school day. 
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cannot be attributed to specific program 
components, and, in this case in particu- 
lar, school coordination efforts appear to 
be an equally minor component of both 
the math and reading studies. 

Finally, a large study evaluated Teach Bal- 
timore, a seven-week summer program 
designed to counter the “summer slide” 
effect^°2 using undergraduate students 
as instructors for students in high-pov- 
erty schools, beginning in kindergarten 
and 1st grade and continuing over three 
summers. The curricula were focused 
primarily on reading instruction and de- 
signed to align with what students were 
learning in their school-year classrooms. 
The evaluators noted that this likely helped 
convey to teachers and principals the po- 
tential benefits of the program, but there 
was not enough detail in the study to de- 
termine the extent of alignment with the 
school year. Instructors also maintained 
communication with their students during 
the school year. 

However, the randomized controlled trial 
found no positive effects on reading that 
included all students assigned to the treat- 
ment group. The authors suggest that this 
result could be partially explained by the 
poor attendance of participating students, 
and they also include a quasi-experimen- 
tal analysis of the average effect on stu- 
dents who attended more frequently.^”^ 
Frequent attendance was defined as hav- 
ing above average attendance (above 39 
percent) for at least two out of the three 
summers. A total of 202 treatment stu- 
dents (46 percent of the original treat- 
ment group) were compared to members 



202. “Summer slide” refers to the effect, broadly 
recognized in the research, by which the achieve- 
ment gap for low-income and minority students 
grows during the summer and stays constant 
while school is in session. See, for example, 
Heyns (1978); Alexander and Entwisle (1996); 
Cooper et al. (1996). 

203. Borman and Dowling (2006). 

204. Ibid. 



of the control group who were weighted 
by their likely predicted probability of 
being a frequent attendee if they had been 
in the treatment group. The analysis 
found statistically significant effects of 
the program for frequent attendees on 
their total reading scores from the Com- 
prehensive Test of Basic Skills, 4th ed., and 
its two subtests: Vocabulary and Reading 

Comprehension. 206 



Inconclusive and cautionary evidence 
for alignment between in-school time 
and OST 

Evaluations of six other after-school pro- 
grams showed no significant positive ac- 
ademic effects, despite efforts by pro- 
grams to coordinate to some degree with 
the in-school instruction. However, the 
amount of coordination in practice in all 
but one of these programs appears to be 
relatively low. The exception is Leap Frog, 
an after-school tutoring program in which 
teachers and tutors have formalized mech- 
anisms for frequent communication. The 
other five programs are the NDP, YS-CARE, 
L.A.’s BEST, 21st CCLC, and SES. 

A small, randomized controlled trial study 
of Leap Erog, an after-school program pro- 
viding one-on-one tutoring and homework 
assistance to 1st and 2nd graders, found 



205. Frequent attendance was predicted based 
on a model that included a baseline measure 
of a student’s regular school-year attendance, 
whether a student moved during the course of the 
study, and the site that a control student would 
have attended if he or she had been in the treat- 
ment group. It is not known whether the frequent 
attendees were similar to the weighted sample of 
control group students in terms of their initial 
reading levels or other characteristics. 

206. Effect sizes were 0.30 for total reading, 0.32 
for vocabulary, and 0.28 for comprehension. 

207. Udell (2003); McKinney (1995); Bissell et al. 
(2002); Goldschmidt, Huang, and Chinen (2007); 
U.S. Department of Education (2003); Dynarski et 
al. (2004); James-Burdumy et al. (2005); Ross et al. 
(2008); Munoz, Potter, and Ross (2008). 
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no academic effects. In the program, 
university student-volunteers tutored for 
one-hour sessions twice a week using work 
sent by the students’ classroom teacher. 
Teachers and tutors communicated using 
a folder, which the tutee carried from 
school to the OST program. Teachers sent 
the tutors assignments or content areas 
to focus on, and tutors were required to 
send weekly comments to the classroom 
teacher about the tutoring sessions. 

A small quasi-experimental study of the 
NDP program, which provided one-on-two 
or one-on-one reading tutoring by under- 
graduate tutors with 2nd-grade students 
after school, found no effects on reading 
compared to a comparison group that re- 
ceived small group activities not related to 
literacy.2^° Although the author reported 
that tutors used techniques that supple- 
mented the school-day instruction, no ad- 
ditional details were offered and coopera- 
tion with the school was not described as 
a key component of the program. 

A large evaluation of the YS-CARE after- 
school program, which targeted Califor- 
nia students with parents transitioning off 
welfare, found positive but insignificant 
findings in the first year.^i^ The analysis 
used matched pairs to measure the effec- 
tiveness of the program. Although YS- 



208. McKinney (1995) had 47 students with pre- 
and posttests in math and 44 in reading. 

209. Ibid. 

210. Udell (2003) had 27 students in the analysis 
sample. The following subtests of the Woodcock 
Johnson-Revised were used to measure reading 
achievement: Word Attack, Letter-Word Identifi- 
cation, and Passage Comprehension. 

211. Ibid. 

212. Bissell et al. (2002). The program reached 
567 students in grades 1 to 5 at 28 schools. 

213. There were no baseline differences in the 
test scores of the treatment and comparison 
groups at the outset of the study, but there were 
some differences in student ethnicity. Also, some 
students in the comparison group reported that 
they did not attend because of responsibilities for 
caring for siblings, and it was not shown whether 



CARE program objectives included rein- 
forcing the school curriculum, the authors 
did not describe the specific activities un- 
dertaken to promote collaboration, nor did 
they report on the actual level of coopera- 
tion implemented except for one survey 
question. 214 Sixty-four percent of surveyed 
staff reported that getting cooperation 
from schools was “not too difficult.”2i3 

In a quasi-experimental study of L.A.’s 
BEST, the authors compared treated stu- 
dents (those attending a minimum of four 
days per month) to two matched com- 
parison groups (those who opted not to 
participate in the same school and those 
in similar schools that did not offer a pro- 
gram) and found no consistent positive 
effects. 21® The program was intended to 
raise academic achievement and reduce 
crime among students in high-crime, high- 
poverty neighborhoods in Los Angeles 
County. Another study used qualitative 
methods to gauge the implementation of 
L.A.’s BEST at six representative sites, iden- 
tified by the main program office . 212 The 
researchers conducted observations, inter- 
views, and focus groups of key stakehold- 
ers. Most principals reported cooperative 
working relationships with OST staff.2i® 
Although participating staff made some 
efforts to collaborate with the school day, 
including attending teacher meetings to 
promote the program, requiring principal 
support before a program was initiated at 
a school, and communicating regularly 
with teachers, these efforts were informal 



the treatment and comparison groups were ini- 
tially equivalent in this way. 

214. Bissell et al. (2002). 

215. Ibid. 

216. Despite the matching attempts, the second 
group (students at non-L.A.’s BEST schools) was 
not comparable in baseline achievement levels 
in math, so the focus is on the analysis with the 
comparison group from nonparticipating stu- 
dents at the same school (Goldschmidt, Huang, 
and Chinen 2007). 

217. Huang et al. (2007). 

218. Ibid. 
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and not necessarily implemented consis- 
tently across sites. 

A national evaluation of the 21st CCLC af- 
ter-school program showed no consistent 
positive academic effects among elemen- 
tary or middle school students. Elemen- 
tary students were randomly assigned to 
either treatment or comparison groups in 
26 centers across 12 districts. Participat- 
ing elementary students reported feeling 
safer than the comparison group students 
reported feeling, but they also experienced 
increased incidents of negative behavior 
during the school day.^^° A matched pair, 
quasi-experimental analysis of the pro- 
gram’s impacts on middle school students 
in 61 centers in 32 districts found similar 
results for behavior, although participat- 
ing students did not report feeling safer 
than control students reported feeling . 221 

The 21st CCLC has awarded grants to sup- 
port after-school programs with an aca- 
demic component (including homework 
help) since 1998. The specifics of the pro- 
gram vary considerably by site. Although, 
on average, the programs were not found to 
be effective at improving academic achieve- 
ment, the study reported that some OST 
coordinators would acquire lists of failing 
students from classroom teachers to iden- 
tify which OST participants needed extra 
attention. Centers reported communicat- 
ing regularly with host schools in setting 
the curricula, goals, and objectives and in 
providing feedback on students. Many prin- 
cipals of host schools also were actively in- 
volved in the planning of the program . 222 

Two quasi-experimental studies of SES in 
various cities in Tennessee and in Louis- 
ville, Kentucky, found overall null effects 
on achievement growth using matched 
pairs to measure the effects of individual 



219. Ibid. 

220. James-Burdumy et al. (2005). 

221. Dynarski et al. (2004). 

222. U.S. Department of Education (2003). 



providers. 223 in the study of SES provision 
in Tennessee, 36 percent of principals re- 
ported that SES providers communicated 
frequently with them, but in a similar 
study in Louisville, Kentucky, none of the 
surveyed principals reported frequent 
communication. Similarly, 34 percent of 
surveyed teachers in Tennessee reported 
frequent contact with SES providers com- 
pared to 7 percent in Louisville. 22 ^ Both 
studies involved relatively small samples 
of principals (50 and 19, respectively) and 
teachers (128 and 56, respectively). 



Supplemental evidence from 
other sources 

Eive other sources provided examples of 
practices for in-school and out-of-school 
alignment even though they either did not 
meet the OST protocol for inclusion or failed 



223. Ross et al. (2008); Munoz, Potter, and Ross 
(2008). 

224. Furthermore, in Louisville, 72 percent of 
teachers reporting having no contact at all with 
SES providers, compared to 21 percent in Ten- 
nessee. Nevertheless, when teachers in Louisville 
were asked whether providers “adapted the tutor- 
ing services to this school’s curriculum,” 7 percent 
strongly agreed and 34 percent agreed. In Tennes- 
see, 30 percent of teachers strongly agreed and 40 
percent agreed. Very similar responses were given 
when asked if providers “integrated the tutoring 
services with classroom learning activities” (43 
percent agreed or strongly agreed in Louisville 
and 68 percent did so in Tennessee) or “aligned 
their services with state and local standards” (41 
percent agreed or strongly agreed in Louisville 
and 71 percent did so in Tennessee). 

A similar survey was done in Virginia but included 
parents, SES providers, and SES coordinators in 
participating school divisions (McKay et al. 2008): 
34 percent of SES coordinators (N = 41) agreed or 
strongly agreed that providers “adapted the tu- 
toring services to this school’s curriculum”; 24 
percent agreed or strongly agreed that provid- 
ers “integrated the tutoring services with class- 
room learning activities”; and 66 percent agreed 
or strongly agreed that providers “aligned their 
services with state and local standards.” Of the 
111 representatives from 16 SES providers, 28 
percent reported frequent communication with 
teachers. 
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to meet WWC standards. A study of the 
level of coordination between classroom in- 
struction and Title 1 supplementary instruc- 
tion found that curricular congruence (e.g., 
both the classroom and the supplemental 
instructor used the same curricular materials 
or content or taught at the same instructional 
level) was correlated with higher reading 

achievement in Ist-grade classrooms. 

The evaluation of The After-School Corpo- 
ration (TASC) showed consistently positive 
effects for math, but not for reading . 227 
Eighty-six percent of principals reported 
that the “after-school program solicits 
input from the principal and teachers on 
skills in which students need help and in- 
corporates these topics into after-school 
activities.”228 Fifty-one percent of princi- 
pals reported that OST instructors “coor- 
dinate homework assistance with class- 
room teachers,” and the same percentage 
reported that the program coordinator 
“serves on a school planning team.”^^^ Al- 
most as frequently, however, principals 
cited the following areas as in need of at- 
tention; “coordination/integration with the 
school curriculum” (41 percent) and “coor- 
dination with the school” (31 percent).23o 

In Project Adelante, a summer program and 
Saturday academy for Hispanic students, 
the project coordinator sent weekly prog- 
ress reports to participating students’ school 



225. Borman (1997); Reisner et al. (2004); Center 
for Applied Linguistics (1994); Leslie (1998); Bouf- 
fard. Little, and Weiss (2006). 

226. Borman (1997). This study was ineligible 
for review because the supplementary instruc- 
tion took place during the school day. Analysis 
was limited to students in classrooms with two 
or more students receiving Title 1 supplemen- 
tary instruction and at least two students not 
receiving it. 

227. Reisner et al. (2004) could not be rated by 
WWC reviewers because it was not clear if the 
treatment and control groups were equivalent in 
achievement levels before the program began. 

228. Reisner et al. (2004). 

229. Ibid. 

230. Ibid. 



counselors, coordinated a parent night and 
invited classroom teachers, and received data 
on English as a Second Language (ESL) stu- 
dents and academic achievement scores from 
the district.231 In a study of an after-school 
tutoring program for rural middle school 
students, tutors met with classroom teach- 
ers prior to each tutoring session to discuss 
the students’ homework and problem areas 
in class. 232 Finally, in a newsletter document- 
ing OST programs’ attempts to create connec- 
tions with the school day, authors cite strat- 
egies including sharing common planning 
periods with school-day teachers, attending 
workshops with teachers, asking teachers to 
help develop programs and deliver instruc- 
tion, and sharing homework completion logs 
with classroom teachers. 2^3 

Recommendation 2. 

Maximize student participation 
and attendance 

Level of evidence: Low 

The panel judged the level of evidence sup- 
porting this recommendation to be low be- 
cause there is no conclusive evidence that 
following the action steps in this recom- 
mendation will lead to higher attendance or 
increased academic achievement. Given the 
voluntary nature of most OST programs and 
other barriers discussed in this recommen- 
dation, regular attendance appears a difficult 
goal for many programs to reach. Some pro- 
grams have devoted considerable resources 
and seem to have made efforts to imple- 
ment the action steps described in this rec- 
ommendation and still have trouble getting 
students to attend regularly (see Table D4). 



231. Center for Applied Linguistics (1994). This 
study was not eligible for review because it did 
not contain a comparison group. 

232. Leslie (1998). This study was not eligible 
for review because it did not contain a compari- 
son group. 

233. Bouffard, Little, and Weiss (2006). This 
newsletter was not eligible for review because 
it was not a study of the effectiveness of these 
strategies. 
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Table D4. Studies of programs cited in recommendation 2 that met WWC standards 
with or without reservations 



Study Details 


Strategies for Recruiting and Promoting Attendance 


Brief Citation 


Program 

Type 


Grade^ 


Average 

Attendance 

Rates'’ 


Recruiting 

Students/ 

Targeting 


Provide 

Snacks/ 

Meals 


Located 
in School/ 
Trans- 
portation 
Provided 


Enrichment 
and Other 
Activities'' 


Monitor 

Attendance 


Incentive 

Systems 


Positive Academic Effects 


KindergARTen, 
Borman, Goetz, 
& Dowling 
(2008) 


Summer 


K 


72% among 
students who 
attended at 
least for 1 day 
(55% overall) 


Teachers, 

principals 


Yes 


Yes 


Recreation, 

cultural 






Early Risers, 
August et al. 
(2001) 


Summer 

and 

school- 

year 

mentoring 


K-2 


50% averaged 
across differ- 
ent program 
components'* 


Teachers 


Yes 


Yes 


Recreation, 

cultural, 

interper- 

sonal 


Yes 




Summer 
Reading Day 
Camp, Schacter 
&Jo (2005) 


Summer 


1 




Target 

schools 


Yes 


Yes 


Recreation, 

cultural 






SMART, Baker, 
Gersten, & 
Keating (2000) 


After- 

school 

tutoring 


1-2 




Teachers 




Yes 








Howard Street 
Tutoring, 
Morris, Shaw, 

& Perney (1990) 


After- 

school 

tutoring 


2-3 




Teachers 


Yes 


Yes 


Recreation 






Fast ForWord, 
Slattery (2003) 


After- 

school 

computer 

tutoring 


3-5 




Students 

reading 

below 

grade 

level 


Yes 


Yes 








CHP, Langberg 
et al. (2006) 


After 

school 


6-7 




Teachers, 

parents, 

student 

testing 


Yes 


Yes 


Recreation 


Yes 




Mixed Academic Effects 


Teach 
Baltimore, 
Borman & 
Dowling (2006) 


Summer 


K-2 


39%, includes 
yearly no- 
shows 


Target 

schools 


Yes 


Yes 


Recreation, 

cultural 


Yes 




Enhanced 
Academic 
Instruction, 
Black et al. (2008) 


After 

school 


2-5 


77% math; 
73% reading 


School 

staff 


Yes 


Yes 


Recreation 


Yes 


Yes 


No Detectable Academic Effects 


L.A.’s BEST, 
Goldschmidt, 
Huang, & 
Chinen (2007) 


After 

school 


K-6 


66% of 
participants 
were active® 


Target 

schools, 

teachers 


Yes 


Yes 


Recreation, 

cultural 


Yes 




21st CCLC 
for elementary 
school students, 
Dynarski et al. 
(2004); U.S. 
Department 
of Education 
(2003) 


After 

school 


K-8 


44% among 
students who 
attended at 
least 1 day' 


Teachers 


Yes 


Yes 


Recreation, 

cultural, 

interper- 

sonal 


Yes 


Yes 



(continued) 
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Table D4. Studies of programs cited in recommendation 2 that met WWC standards 
with or without reservations (continued) 



Study Details 


Strategies for Recruiting and Promoting Attendance 


Brief Citation 


Program 

Type 


Grade“ 


Average 

Attendance 

Rates'’ 


Recruiting 

Students/ 

Targeting 


Provide 

Snacks/ 

Meals 


Located 
in School/ 
Trans- 
portation 
Provided 


Enrichment 
and Other 
Activities'^ 


Monitor 

Attendance 


Incentive 

Systems 


No Detectable Academic Effects (continued) 


Leap Frog, 
McKinney 
(1995) 


After- 

school 

tutoring 


1-2 




Teachers, 

principals 


Yes 


Yes 


Recreation, 

cultural 






YS-CARE, 
Bissell et al. 
(2002) 


After 

school 


1-5 






Yes 


Yes 


Recreation, 

cultural, 

interper- 

sonal 






NDP, Udell 
(2003) 


After- 

school 

tutoring 


2 




School 

staff 


Yes 


Yes 




Yes 


Yes 



a. For summer programs, refers to the grade participating students exited prior to the commencement of the summer program. 

b. Studies with blank cells did not report any information on attendance rates. 

c. Examples of recreational activities include soccer, karate, and dance. Cultural activities include music, arts, and field trips. Interpersonal activities 
include conflict management and relationship building. 

d. August et al. (2001) calculated summer school attendance and family program attendance as the percentage of days attended out of total days of- 
fered over the course of two years. They took a simple average of these two attendance rates with a third “rate,” which they used to measure the 
“percentage” of FLEX contact time each family received. FEEX contact time was meant to provide preventative case management as needed. Actual 
contact time ranged from 20 minutes to more than 5 hours total, excluding the initial 2-hour home visit to all participants. A full dosage of FLEX 
time was capped at 2 hours (based on the assumption of two more 1-hour contacts); 22 percent of families received the full dosage of FLEX time. 

e. Active participation is defined as attending the program more than 36 days a year. This participation rate is from our calculations using the informa- 
tion in Table 3 in Goldschmidt, Fluang, and Chinen (2007) from 1994 to 1997, the years included in their impact study. Their sample was restricted 
to schools with more than 20 students with program attendance information. 

f. Estimated based on information in Table 11.1 in Dynarski et al. (2004). Sample size is 980 students who attended an elementary school center for at 
least one day. 
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However, given that actual attendance is 
a precursor to an OST program’s promot- 
ing student learning, the panel believes 
it is particularly important for programs 
to enhance their efforts to get students in 
the door. 



Summary of evidence 

The importance of emphasizing partici- 
pation has been pointed out by many ex- 
perts in OST.234 Although it seems logical 
that students need to attend to receive 
the benefits of a program, no rigorous 
evidence demonstrates that the steps rec- 
ommended here will lead to increased par- 
ticipation, and limited evidence indicates 
that academic achievement is increased 
through more exposure to OST programs. 
A meta-analysis of 53 OST studies by Lauer 
et al. (2004) found larger effect sizes in 
both math and reading for interventions 
that consisted of at least 45 hours of pro- 
gramming. The panel believes that if a 
program is aligned academically with the 
school day (recommendation 1), provides 
engaging learning experiences (recom- 
mendation 3), and adapts instruction to 
individuals and groups (recommendation 
4), greater exposure to that program will 
yield higher academic achievement. 

Since a student’s actual program atten- 
dance, as a percentage of hours of pro- 
gramming offered, is likely to be correlated 
with unobservable characteristics such as 
motivation or family circumstances, it 
is not advisable to draw causal conclu- 
sions from most studies on the relation- 
ship between attendance and outcomes. 
Four evaluations of programs met WWC 
standards with or without reservation for 



234. Cooper et al. (2000); Granger and Kane 
(2004); Lauver, Little, and Weiss (2004). 

235. The meta-analysis also found that, on aver- 
age, programs with high durations (more than 100 
hours for math and 210 hours for reading) did not 
have effects significantly different from zero. 



their academic impact studies and also 
looked at the possible relation between 
amount of program attendance and aca- 
demic achievement. Only one found a 
positive correlation between higher atten- 
dance and greater program effects. 

The other evidence that influences this rec- 
ommendation is less direct. Fourteen pro- 
grams had studies that discussed practices 
similar to the action steps recommended 
by the panel (such as using teachers to 
recruit students, locating within schools, 
offering snacks, and including enrichment 
activities) and also met WWC standards 
with or without reservations. Of the 
14, only 6 reported some information on 
attendance rates. Even when programs 
reported attendance, it was not possible 
to isolate which, if any, components of 
the program affected attendance, forcing 
the panel to use its judgment regarding 
which practices contributed to increased 
attendance and improved academic out- 
comes. In terms of overall academic ef- 
fects, 7 of the 14 programs showed positive 



236. Teach Baltimore — Borman and Dowl- 
ing (2006); Early Risers — August et al. (2001); 
L.A.’s BEST — Goldschmidt, Huang, and Chinen 
(2007); 21st CCLC — U.S. Department of Educa- 
tion (2003). 

237. Teach Baltimore — Borman and Dowling 
(2006). 

238. CHP — Langberg et al. (2006); East EorWord — 
Slattery (2003); Howard Street Tutoring — Morris, 
Shaw, and Perney (1990); SMART— Baker, Gersten, 
and Keating (2000); Summer Reading Day Camp — 
Schacter and Jo (2005); KindergARTen — Borman, 
Goetz, and Dowling (2008); Early Risers — August 
et al. (2001); Enhanced Academic Instruction — 
Black et al. (2008); Teach Baltimore — Borman 
and Dowling (2006); L.A.’s BEST — Goldschmidt, 
Huang, and Chinen (2007); YS-CARE — Bissell et 
al. (2002); 21st CCEC — U.S. Department of Educa- 
tion (2003); Eeap Erog — McKinney (1995); NDP — 
Udell (2003). 

239. Early Risers — August et al. (2001); Kinder- 
gARTen — Borman, Goetz, and Dowling (2008); 
Enhanced Academic Instruction — Black et al. 
(2008); Teach Baltimore — Borman and Dowling 
(2006); E.A.’s BEST — Goldschmidt, Huang, and 
Chinen (2007); 21st CCEC — U.S. Department of 
Education (2003). 



60 



APPENDIX D. TECHNICAL INFORMATION ON THE STUDIES 



effects, 240 2 showed mixed effects, and 
5 others showed no effects. 2^2 Despite the 
lack of consistent evidence linking the 
panel’s suggestions to increased academic 
achievement, the panel believes these rec- 
ommendations, faithfully implemented 
and taking into consideration the unique 
constraints and student populations of 
each program, can increase student at- 
tendance and, therefore, contribute to 
achievement gains. 

Only one study provided direct evidence 
on increasing attendance. Black et al. 
(2008) randomly assigned students to ei- 
ther a less-structured business-as-usual 
after-school program or an enhanced math 
or reading program that included moni- 
toring and incentive systems to increase 
attendance. Students in the enhanced pro- 
gram attended significantly more days 
than did control students. 2^2 



240. CHP — Langberg et al. (2006); Fast ForWord — 
Slattery (2003); Floward Street Tutoring — Morris, 
Shaw, and Perney (1990); SMART— Baker, Gersten, 
and Keating (2000); Summer Reading Day Camp — 
Schacter and Jo (2005); KindergARTen — Borman, 
Goetz, and Dowling (2008); Early Risers — August 
et al. (2001). 

241. Teach Baltimore — Borman and Dowling 
(2006); Enhanced Academic Instruction — Black 
et al. (2008). 

242. L.A.’s BEST — Goldschmidt, Huang, and 
Chinen (2007); YS-CARE— Bissell et al. (2002); 21st 
CCLC — U.S. Department of Education (2003); Leap 
Erog— McKinney (1995); NDP— Udell (2003). 

243. Students in their math treatment group at- 
tended 12 days more over the course of the year 
compared to students in the regular program 
who attended 61 days on average. In their read- 
ing study, students attended 7 more days com- 
pared to the treatment group, which attended 64 
days. Both of these differences are statistically 
significant at the 5 percent level and represent a 
20 percent increase and 11 percent increase in 
days attended, respectively. 



Four studies on the effects of attendance 
on achievement 

As discussed in recommendation 1, a ran- 
domized controlled trial of Teach Balti- 
more, a summer school program, found no 
positive effects on reading for all partici- 
pants, but the quasi-experimental evalu- 
ation found positive effects on students 
who attended frequently. 2^4 Although it is 
encouraging that there were effects among 
the frequent attendees, the fact that the av- 
erage attendance rate across all three sum- 
mers was only 39 percent and that only 46 
percent of the original treatment group 
could be classified as frequent attendees is 
an important cautionary note as the Teach 
Baltimore program does include some ele- 
ments that the panel recommends to en- 
courage attendance. For example, staff in 
the program served lunch to all program 
participants during the seven-week sum- 
mer program. After lunch, there was time 
for sports and enrichment activities such 
as music and drama and weekly field trips 
to museums and cultural events. 

It also is important to note that three stud- 
ies did not find higher impacts on aca- 
demic achievement for higher attendees. 
The Early Risers program did find positive 
academic impacts overall, but students 
who were classified as receiving less than 
half of the “program dosage” improved 
at the same rate as those who received 
half or more. 245 The other two programs, 
21st CCLC and L.A.’s BEST, failed to find 
academic impacts either overall or for 
higher attendees. The evaluations of both 
L.A.’s BEST and Early Risers used regres- 
sion approaches to control for observ- 
able differences between high attendees 
and low attendees. However, unobserved 



244. Borman and Dowling (2006). Erequent at- 
tendance was defined has having above average 
attendance (above 39 percent) for at least two out 
of the three summers. The average attendance 
rate for the program includes students who did 
not participate for an entire summer. 

245. August et al. (2001). 
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characteristics may be driving the null 
results, and their analysis should not be 
interpreted as causal. 

In the second- and third-year reports 
from the national study of 21st CCLC, 
fixed effects analysis was used to evalu- 
ate whether more frequent attendance 
had a positive impact on a wide variety of 
outcomes. This analysis uses two years 
of data on the same students; therefore, 
time-invariant unobservable characteris- 
tics cannot be driving the results. It tests 
whether the changes in an individual stu- 
dent’s attendance are related to changes in 
outcomes; therefore, it depends on there 
being sufficient variation in both atten- 
dance and outcomes. Only small insig- 
nificant marginal effects were found on 
year-end grades for both the elementary^^s 
and middle school samples. 

Goldschmidt, Huang, and Chinen (2007) 
found no positive effects for active par- 
ticipation in an L.A.’s BEST after-school 
program on achievement scores or growth 
in either reading or math, measured at the 



246. Dynarski et al. (2004); James-Burdumy et 
al. (2005). 

247. However, time-varying ones still could be 
responsible. The first-year report of 21st CCLC 
(Department of Education 2003) considered using 
statistical techniques with their middle school 
sample to account for the possibility that there 
could be unobservable factors responsible for 
influencing both attendance and academic out- 
comes, but they could not find a suitable instru- 
mental variable that was both related to atten- 
dance but not Independently related to academic 
outcomes. Mother’s employment status was not 
correlated with program attendance. 

248. James-Burdumy et al. (2005), p. 80. 

249. Ibid., p. 83. 

250. Dynarski et al. (2004), p. 131. 

251. Nonlinear models were used to calculate 
the marginal effect of attending the program for 
10 more days on both an average student with 
a low level of attendance (10 days a year) and a 
student who attended for 30 days. Neither case 
showed significant improvement in year-end 
grades in any subject whether in elementary or 
middle school. 



beginning of middle school after students 
had spent up to five years participating 
in L.A.’s BEST in elementary school. 

In hierarchical linear models similar to 
those used to estimate the overall treat- 
ment effects, the authors also controlled 
for exposure to L.A.’s BEST (years of pro- 
gram attendance) and intensity of expo- 
sure (natural log of daily attendance rate in 
years attended). The estimated coefficients 
on these variables in all four models (lev- 
els and growth in both math and reading) 
were always small and never significantly 
different from zero. 

It appears that the authors also examined 
the correlations of academic outcomes with 
either program exposure or intensity using 
only program participants. The magnitude 
of effects is not reported, and, overall, the 
results appear mixed. Both exposure and 
intensity are reported to be insignificant 
for reading achievement level at the begin- 
ning of middle school but marginally sig- 
nificant for reading growth. Exposure is 
not found to have a significant effect on ei- 
ther math levels or growth, and intensity is 
reported to be significantly related to math 
levels but not to math growth. 



Indirect evidence on recruitment and 
ways to encourage attendance 

Other studies provided supplementary 
examples of how to promote attendance 



252. Active participation was defined as attend- 
ing at least 36 days a year. From 1994 to 1997, 66 
percent of students who ever attended an L.A.’s 
BEST program during the course of a school year 
attended at least 36 days that year. The average 
number of days attended of all students from 
1994 to 1997 was 72. 

253. They ran the same models using a second 
comparison group consisting of matched stu- 
dents at non-E.A.’s BEST schools. However, de- 
spite the matching attempts, the second group 
was not comparable in baseline achievement 
levels in math, so we focus on the analysis with 
the first group. 

254. p<.10 
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at OST programs. Examples that support 
the panel’s recommendations appear in 
14 programs that had studies that met 
WWC standards with or without reser- 
vations. Seven of those programs had 
positive effects on academics, two had 
mixed effects, and five had no effect on 
academic achievement.^^® In general, this 
information should be interpreted cau- 
tiously; there is no direct evidence that 
the practices used by these programs were 
directly linked to improvements in either 
attendance or academic achievement, and 
many of the practices were observed both 
in programs that produced positive aca- 
demic effects and in those that did not. 

Table D4 compares several of the panel’s 
suggestions for promoting attendance in 
OST programs to the components dis- 
cussed by evaluators in studies of the 12 
programs that provided evidence for this 
recommendation. Of the 9 studies that 
found positive or mixed academic treat- 
ment effects, 6 used teachers, principals, 
or school staff to recruit students. Two 
programs targeted schools with particular 
characteristics, and one randomly chose 
among all students reading below grade 



255. Langberg et al. (2006); Slattery (2003); Mor- 
ris, Shaw, and Perney (1990); Baker, Gersten, and 
Keating (2000); Schacter and Jo (2005); Borman, 
Goetz, and Dowling (2008); August et al. (2001); 
Black et al. (2008); Borman and Dowling (2006); 
Goldschmidt, Huang, and Chinen (2007); Bissell 
et al. (2002); U.S. Department of Education (2003); 
McKinney (1995); Udell (2003). 

256. Langberg et al. (2006); Slattery (2003); Mor- 
ris, Shaw, and Perney (1990); Baker, Gersten, and 
Keating (2000); Schacter and Jo (2005); Borman, 
Goetz, and Dowling (2008); August et al. (2001). 

257. Black et al. (2008); Borman and Dowling 
(2006). 

258. Goldschmidt, Huang, and Chinen (2007); 
Bissell et al. (2002); U.S. Department of Education 
(2003); McKinney (1995); Udell (2003). 

259. Langberg et al. (2006); Morris, Shaw, and 
Perney (1990); Baker, Gersten, and Keating (2000); 
Borman, Goetz, and Dowling (2008); August et al. 
(2001); Black et al. (2008). 

260. Schacter and Jo (2005); Borman and Dowl- 
ing (2006). 



level. All but one^®^ provided snacks 
or meals, and all either took place in the 
students’ school or provided transporta- 
tion to students. 263 All but two^®^ included 
some recreational component for partici- 
pating students. Three also provided cul- 
tural components, 263 and one provided 
interpersonal development activities for 
students. 266 Four of the programs moni- 
tored attendance, 267 and one had an incen- 
tive system to promote attendance. 26® 

In studies of the five programs that demon- 
strated no treatment effects on academic 
achievement, researchers highlighted 
many of the same components. 269 All but 
one program27o relied on teachers, princi- 
pals, and school staff to identify students 
for participation. All provided a snack and 
were either located in the students’ schools 
or provided transportation to the program 
site. 271 Four of the programs provided 
recreational and cultural activities , 272 and 
two provided interpersonal activities. 273 



261. Slattery (2003). 

262. Baker, Gersten, and Keating (2000). 

263. Langberg et al. (2006); Morris, Shaw, and 
Perney (1990); Baker, Gersten, and Keating (2000); 
Schacter and Jo (2005); August et al. (2001); Black 
et al. (2008); Borman and Dowling (2006); Borman, 
Goetz, and Dowling (2008); Slattery (2003). 

264. Slattery (2003); Baker, Gersten, and Keat- 
ing (2000). 

265. Schacter and Jo (2005); August et al. (2001); 
Borman and Dowling (2006). 

266. August et al. (2001). 

267. Langberg et al. (2006); August et al. (2001); 
Black et al. (2008); Borman and Dowling (2006). 

268. Black et al. (2008). 

269. Treatment effects refer to the overall aca- 
demic program impact, not to whether the impact 
varied with level of attendance. 

270. Bissell et al. (2002). 

271. Goldschmidt, Huang, and Chinen (2007); 
Bissell et al. (2002); U.S. Department of Education 
(2003); McKinney (1995); Udell (2003). 

272. Goldschmidt, Huang, and Chinen (2007); 
Bissell et al. (2002); U.S. Department of Education 
(2003); McKinney (1995). 

273. Bissell et al. (2002); U.S. Department of Edu- 
cation (2003). 
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Three studies indicated that program staff 
monitored attendance, and two docu- 
mented providing incentives to increase 
attendance. 

Overall, only 6 of the 14 programs provide 
some information on attendance rates, and 
the information they provide is not all di- 
rectly comparable. 276 Together, the studies 
suggest that given the evidence base, it is 
not possible to attribute causal effects on 
attendance to the strategies that the panel 
recommended. Furthermore, there is little 
evidence in the studies to suggest that the 
programs that show positive academic ef- 
fects are doing something fundamentally 
different to increase program attendance. 
Nevertheless, the studies that report at- 
tendance rates highlight how difficult it 
is for programs to achieve full participa- 
tion, implying the need for a recommen- 
dation from the panel on encouraging 
participation. 



Supplemental evidence from 
other sources 

Four additional studies that did not meet 
WWC standards looked at interventions that 
incorporated some parts of the action steps 
from this recommendation. The Building 
Education Leaders for Life (BELL) program, 
which was a summer program for students 
in grades 1-7, was purposefully located on 
school campuses. 277 The BELL organizers 



274. Goldschmidt, Huang, and Chinen (2007); U.S. 
Department of Education (2003); Udell (2003). 

275. U.S. Department of Education (2003); Udell 
(2003). 

276. Borman, Goetz, and Dowling (2008); Au- 
gust et al. (2001); Black et al. (2008); Borman and 
Dowling (2006); Goldschmidt, Huang, and Chinen 
(2007); U.S. Department of Education (2003). 

277. Capizzano et al. (2007). The randomized 
controlled trial evaluation of BELL by Chaplin 
and Capizzano (2006) did not meet WWC stan- 
dards because it suffered from attrition greater 
than 20 percent and had not measured the initial 
achievement levels and could not demonstrate 
that the treatment and control groups were 



reported that obtaining and maintaining 
the school space was one of the most chal- 
lenging aspects of running the program, yet 
each summer they were located in schools. 
The after-school tutoring projects run by 
TASC were similarly located in the schools 
attended by participants. 27^ The evaluation 
of the TASC program found that projects 
also targeted the students most likely to 
benefit from the program, encouraged high 
student attendance, and achieved a median 
attendance rate of 85 percent. 

Project Adelante provided supervised 
transportation during the summer to the 
college campus where tutoring, counsel- 
ing, and mentoring services were provided 
to Latino middle and high school stu- 
dents. 279 Students in the target population 
were recruited from the regular school day 
by school staff. Program staff also were 
actively involved in promoting student at- 
tendance, first by requiring parents to sign 
a contract stating that they would support 
attendance, and then by following up with 
students who missed sessions. 

The Broward County Boys and Girls Club, 
although not a school-based program, also 
promoted program attendance by provid- 
ing incentives to students in the form of 
points that could be redeemed for field 
trips or events or used toward the pur- 
chase of books and supplies. 26° 



equivalent initially. Furthermore, the posttest 
achievement tests for students in the control 
group were administered at a different date than 
the treatment group and are, therefore, not di- 
rectly comparable. 

278. Reisner et al. (2004). This study could not 
be rated by WWC reviewers because it was not 
clear if the treatment and control groups were 
equivalent in achievement levels before the pro- 
gram began. 

279. Center for Applied Linguistics (1994). This 
study did not meet standards because it was a 
qualitative evaluation of the program. 

280. Brown (2002). This study was ineligible for 
review because it did not use a comparison group 
in the analysis. 
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In addition to these studies of interven- 
tions, a national survey by Public Agenda 
of a random sample of students and par- 
ents of school-age children explored what 
students do when they are not in school, 
and what students and parents want from 
out-of-school time.^®^ The survey found 
that student participation is affected by 
access to and convenience of programs, 
as well as by the attractiveness of the ser- 
vices provided. 

Recommendation 3. 

Adapt instruction to individual and 
small group needs 

Level of evidence: Moderate 

The panel judged the level of evidence 
supporting this recommendation to be 
moderate. Learning environments that are 
adaptive to individual and small group 
needs are widely believed to be effective in 
fostering achievement. Within the con- 
text of OST, however, the literature is not 
definitive, and, in general, positive effects 
cannot be directly attributed to the use of 
the strategies outlined in this recommen- 
dation. Looking more closely at the actual 
implementation of practices related to this 
recommendation, however, there does ap- 
pear to be a pattern of positive academic 
effects associated with programs that 
more closely correspond to this recom- 
mendation. Therefore, the panel believes 
OST programs can be more successful if 
they attempt to understand the academic 
needs of the students they serve and adapt 
their programs to those needs. 

Summary of evidence 

Of the 15 programs related to this recom- 
mendation with evaluations that met WWC 



281. Duffett et al. (2004). 

282. See Slavin (2006) in general and Lauer et al. 
(2004) for OST programs in particular. 



standards with or without reservations, 

5 were judged to be in close correspon- 
dence with more than one aspect of this 
recommendation (see Table D5).^®^ Of these, 
4 had evaluations that found positive ef- 
fects on academic achievement,^®^ and 
one had mixed but potentially encourag- 
ing effects.^®® Of the other 10 programs, 

6 still showed positive or mixed effects on 
academics.^®® 



283. KindergARTen — Borman, Goetz, and Dowling 
(2008); Early Risers — August et al. (2001); Sum- 
mer Reading Day Camp — Schacter and Jo (2005); 
Experience Corps — Meier and Invernizzi (2001); 
SMART— Baker, Gersten, and Keating (2000); How- 
ard Street Tutoring — Morris, Shaw, and Perney 
(1990); East EorWord — Slattery (2003); CHP — Lang- 
berg et al. (2006); Teach Baltimore — Borman and 
Dowling (2006); Enhanced Academic Instruction — 
Black et al. (2008); Chicago Summer Bridge — ^Jacob 
and Lefgren (2004); Eeap Erog — McKinney (1995); 
NDP— Udell (2003); 21st CCEC— U.S. Department 
of Education (2003); 21st CCEC — Dynarski et al. 
(2004); 21st CCEC— James-Burdumy et al. (2005); 
SES— McKay et al. (2008); SES— Ross et al. (2008); 
SES — Munoz, Potter, and Ross (2008). 

284. Early Risers — August et al. (2001); Howard 
Street Tutoring — Morris, Shaw, and Perney (1990); 
Fast EorWord — Slattery (2003); CHP — Langberg 
et al. (2006); Enhanced Academic Instruction — 
Black et al. (2008). 

285. Howard Street Tutoring — Morris, Shaw, and 
Perney (1990); Early Risers — August et al. (2001); 
Fast EorWord — Slattery (2003); CHP — Langberg 
et al. (2006). 

286. Enhanced Academic Instruction — Black et 
al. (2008). 

287. KindergARTen — Borman, Goetz, and Dowl- 
ing (2008); Summer Reading Day Camp — Schacter 
and Jo (2005); Experience Corps — Meier and In- 
vernizzi (2001); SMART — Baker, Gersten, and 
Keating (2000); Teach Baltimore — Borman and 
Dowling (2006); Chicago Summer Bridge — ^Jacob 
and Lefgren (2004); 21st CCEC — U.S. Department 
of Education (2003); 21st CCEC — Dynarski et al. 
(2004); 21st CCEC — ^James-Burdumy et al. (2005); 
SES— McKay et al. (2008); SES— Ross et al. (2008); 
SES — Munoz, Potter, and Ross (2008); Leap Frog — 
McKinney (1995); NDP— Udell (2003). 

288. KindergARTen — Borman, Goetz, and Dowl- 
ing (2008); Summer Reading Day Camp — Schacter 
and Jo (2005); Experience Corps — Meier and In- 
vernizzi (2001); SMART — Baker, Gersten, and 
Keating (2000); Teach Baltimore — Borman and 
Dowling (2006); Chicago Summer Bridge — ^Jacob 
and Lefgren (2004). 
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Table D5. Studies of programs cited in recommendation 3 that met WWC standards 
with or without reservations 



Program and Study Details 


Strategies 


Brief Citation 


Program 

Type 


Grades 

Studied^ 


Level of 
Relevance*’ 


Use 

Assessments 
to Inform 
Teaching 


One-on-One/ 
Small Group 


Professional 

Development 


Type of 
Staff Used 


Positive Academic Effects 


KindergARTen, 
Borman, Goetz, 

& Dowling (2008) 


Summer 


K 


V 


Yes 




Initial and 
ongoing 


Certified teach- 
ers and college 
student interns 


Summer Reading Day 
Camp, Schacter & Jo 
(2005) 


Summer 


1 


V 




Small group 
for some 
activities 




Certified 

teachers 


Early Risers, August 
et al. (2001) 


Summer and 
school-year 
mentoring 


K-2 


vv 


Yes 


Small group 


Initial and 
ongoing 




Experience Corps/ 
Book Buddies, Meier 
& Invernizzi (2001) 


After-school 

tutoring 


1 


V 


Yes 


One-on-one 




Volunteers 


SMART, Baker, Gersten, 
& Keating (2000) 


After-school 

tutoring 


1-2 


V 




One-on-one 




Volunteers 


Howard Street 
Tutoring, Morris, Shaw, 
& Perney (1990) 


After-school 

tutoring 


2-3 


w 


Yes 


One-on-one 


Ongoing 


Volunteers 


Fast ForWord, Slattery 
(2003) 


After-school 

computer 

tutoring 


3-5 


w 


Yes 


One-on-one 

computer 

work 




Certified 

teachers 


CHP, Langberg et al. 
(2006) 


After school 


6-7 


w 




Small group 
for some 
activities 


Initial 


Certified teach- 
ers and CHP 
specialists 


Mixed Academic Effects 


Teach Baltimore, 
Borman & Dowling 
(2006) 


Summer 


K-2 


V 


Yes 


Small group 


Initial 


College 

students 


Chicago Summer 
Bridge, Jacob & Lef- 
gren (2004) 


Summer 


3, 6 


V 








Certified 

teachers 


Enhanced Academic 
Instruction, Black et 
al. (2008) 


After school 


2-5 


w 


Yes 


Small group 


Initial and 
ongoing 


Certified 

teachers 


No Detectable Academic Effects 


SES, McKay et al. 
(2008); Ross et al. 
(2008); Munoz, Potter, 
& Ross (2008) 


After-school 

tutoring 


K-12c 


V 




One-on-one 




Varies 


Leap Frog, McKinney 
(1995) 


After-school 

tutoring 


1-2 


V 




One-on-one 


Initial 


College 
students and 
volunteers 


NDP, Udell (2003) 


After-school 

tutoring 


2 


V 


Yes 


One-on-one 


Initial and 
ongoing 


College 

students 


21st CCLC, U.S. 
Department of Educa- 
tion (2003); Dynarski 
et al. (2004); James- 
Burdumy et al. (2005) 


After school 


K-8 


V 




Varies from 
small to large 
groups 




Varies 



a. For summer programs, refers to the grade participating students exited prior to the commencement of the summer program. 

b. indicates relevance of recommendation to program operation. VV” indicates high relevance; V” indicates lower relevance. 

c. Grades included in SES could be any from kindergarten to 12th grade. Typically, grades included in state or district evaluations are those that are 
more often tested statewide with tests that are comparable across grade levels, usually 3rd through 8th. 
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Of the four programs with lower levels 
of relevance and without detectable ef- 
fects on academic achievement, two de- 
serve special mention because they are 
the two major sources of federal funding 
for academically focused OST programs: 
21st CCLC and SES, which are mandated 
as part of NCLB.^ss The national study of 
21st CCLC programs found no positive 
academic effects for either elementary or 
middle school students. However, im- 
plementation varied widely, and although 
some programs reported tutoring in small 
groups with fewer than 10 students, most 
programs did not seem to be providing 
direct or adaptive instruction that was 
geared to the individual needs of all their 
students. Although SES is a newer program 
relative to 21st CCLC and less often stud- 
ied, it appears to be implemented typi- 
cally as one-on-one or small group tutor- 
ing (again implementation varies widely), 
and the results from the states that have 
attempted to evaluate the effects of SES, as 
mandated by law, do not show significant 
impacts on state assessments. 

In summary, the evidence demonstrates 
positive effects associated with a total of 
eight programs that adapted instruction 
to individual and small groups to some 
degree^®^ and mixed effects in three other 
programs;^^^ however, because adapting 
instruction was always a component of a 



289. Zimmer et al. (2007). 

290. U.S. Department of Education (2003); Dy- 
narski et al. (2004); James-Burdumy, Dynarski, 
and Deke (2007). 

291. Tennessee — Ross et al. (2008); Kentucky — 
Munoz, Potter, and Ross (2008); Virginia — McKay 
et al. (2008). 

292. KindergARTen — Borman, Goetz, and Dowl- 
ing (2008); Early Risers — August (2001); Summer 
Reading Day Camp — Schacter and Jo (2005); Ex- 
perience Corps — Meier and Invernlzzl (2001); 
SMART — Baker, Gersten, and Keating (2000); 
Howard Street Tutoring — Morris, Shaw, and Per- 
ney (1990); Past PorWord — Slattery (2003); CHP — 
Eangberg et al. (2006). 

293. Enhanced Academic Instruction — Black et 
al. (2008); Teach Baltimore — Borman and Dowl- 



multicomponent intervention and because 
adapting instruction did not consistently 
demonstrate significant positive effects 
across every study reviewed, the panel 
is cautious and acknowledges that the 
level of evidence is moderate. 



Positive evidence for individualizing 
instruction in OST programs 

The eight randomized controlled studies 
of OST programs that included individu- 
alized instruction and showed positive 
effects looked at after-school study skills 
programs, mentoring programs, summer 
programs, and literacy programs (either 
one-on-one with a tutor or with a com- 
puter), or some combination. The grades 
studied were elementary in all programs 
except one, 296 and six out of the eight el- 
ementary school programs were for grades 
K-3. The sample sizes for all of these stud- 
ies were relatively small; all but two of the 
nine had fewer than 100 students in their 
total analysis sample. 

The four programs judged to be in higher 
correspondence include two after-school 
tutoring programs (Howard Street Tutoring 



ing (2006); Chicago Summer Bridge — ^Jacob and 
Eefgren (2004). 

294. Eeap Prog — McKinney (1995); NDP — Udell 
(2003); 21st CCLC — U.S. Department of Educa- 
tion (2003); 21st CCLC — Dynarski et al. (2004); 
21st CCLC — ^james-Burdumy, Dynarski, and Deke 
(2007); SES— McKay et al. (2008); SES— Ross et al. 
(2008); SES — Munoz, Potter, and Ross (2008). 

295. KindergARTen — Borman, Goetz, and Dowl- 
ing (2008); Early Risers — August et al. (2001); 
Summer Reading Day Camp — Schacter and Jo 
(2005); Experience Corps — Meier and Invernizzi 
(2001); SMART — Baker, Gersten, and Keating 
(2000); Howard Street Tutoring — Morris, Shaw, 
and Perney (1990); Past PorWord — Slattery (2003); 
CHP — Eangberg et al. (2006). 

296. Eangberg et al. (2006). 

297. Schacter and Jo (2005) had 118 students 
in their first follow-up of participants in a sum- 
mer reading day camp, and August et al. (2001) 
had 201 students in their summer program with 
school-year mentoring. 
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and Fast ForWord), an after-school study 
skills program (CHP), and a summer pro- 
gram with mentoring. August et al. (2001) 
studied Early Risers, a summer school 
and school-year mentoring program for 
1st and 2nd graders. Program staff mem- 
bers monitored the academic, behavioral, 
and social progress of each student. They 
assessed students after each session by 
rating aspects of their performance such 
as engagement and task completion. Staff 
were trained to do the student assessment 
and monitoring required by Early Risers 
through preprogram training, manuals, 
and ongoing support. The study demon- 
strated significant increased academic 
competence for the treatment group rela- 
tive to a comparison group. 

Langberg et al. (2006) studied CHP, which 
provided academic remediation in small 
groups for 6th and 7th graders with a com- 
bination of learning and behavior prob- 
lems. CHP limited class sizes to 12 students 
per instructor but also broke classes into 
smaller groups of 3 students for skill de- 
velopment exercises. The staff members 
were provided with training and manuals 
to guide their instruction. The program 
showed no statistically significant effects 
on teacher ratings of academic progress 
because of the small sample size, but the 
effect size (calculated using Hedges’s 
a difference-in-differences method) was 
positive at 0.45.^^^ In order to acknowledge 
meaningful effects regardless of sample 
size, the panel followed WWC guidelines 
and considered a positive statistically sig- 
nificant effect or an effect size greater than 
0.25 as an indicator of positive effects. 

In the Howard Street tutoring program, 
instructors constantly adapted the pace 
and content of instruction with their 



298. See Appendix B of WWC Procedures 
and Standards Handbook, http://www.ies. 
ed.gov/ncee/wwc/references/idocviewer/Doc. 
aspx?docId=19&tocId=8. 

299. Langberg et al. (2006). 



assessment of the student’s current skill 
level, advancing or reviewing depending 
on each student’s development. further- 
more, all volunteer tutors were given con- 
tinued support via ongoing monitoring by 
the reading specialist supervisor, who also 
planned each student’s lessons individu- 
ally. The Morris, Shaw, and Perney (1990) 
evaluation included five literacy measures. 
After adjusting for multiple comparisons, 
only on reading Basal Passages did the 
treatment group do significantly better 
than the control group on the posttest, 
but effect sizes were greater than 0.25 on 
all five posttest measures. 

finally, among the higher correspondence 
programs, Slattery (2003) studied the East 
forWord computer program in an after- 
school context that was chosen by the 
school so that students in need of extra 
help would not miss any instruction during 
the school day. East EorWord is a computer 
program specifically designed to monitor 
individual performance on a daily basis 
and to modify the activities suggested to 
each student accordingly. Slattery (2003) 
found large positive significant effect sizes 
for both literacy measures that she used. 

The main reason why four of these pro- 
grams were judged to be in lower corre- 
spondence was a lack of detail in the evalu- 
ations describing these types of practices, 
for instance, in two cases, the programs 
used one-on-one tutoring, but there is no 
indication if tutors were trained to do for- 
mative assessments of their students and 
to use these assessments to guide their 
instruction. Similarly, in two summer 
programs with small groups, it simply was 
not known how adaptive the programs 
were in their actual implementation. 



300. Morris, Shaw, and Perney (1990). 

301. Experience Corps — Meier and Invernizzi 
(2001); SMART — Baker, Gersten, and Keating 
( 2000 ). 

302. Schacter and Jo (2005); Borman, Goetz, and 
Dowling (2008). 
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Nevertheless, they may have been, and it 
is notable that all the programs detailed 
in this section did find positive effects on 
some measures of academic achievement. 

The Meier and Invernizzi (2001) study of 
Experience Corps and the Baker, Gersten, 
and Keating (2000) evaluation of SMART 
were both studies of after-school one-on- 
one tutoring programs in reading with 
1st graders. In the Experience Corps Book 
Buddies program, tutoring sessions were 
designed to adapt to the tutee’s individual 
learning pace and needs and to evolve with 
the progress in their learning.^^^ After just 
one semester of tutoring (approximately 
40 sessions lasting 45 minutes each), the 
first treatment group significantly out- 
performed the control group on early lit- 
eracy measures. The SMART program 
in Oregon is a two-year, one-on-one tutor- 
ing program in reading. The tutors were 
volunteers and were given minimal initial 
training and usually received very little or 
no ongoing support. The authors reported 
positive academic effects on various read- 
ing measures, but after applying WWC 
standards for multiple comparisons, these 
effects were no longer statistically signifi- 
cant.^®^ However, their effect sizes were 
larger than 0.25; therefore, the impacts 
are considered substantively important 
following WWC guidelines. 



303. Meier and Invernizzi (2001). 

304. In the next semester, the groups switched 
places and the control group received 40 sessions 
of tutoring. By June, control group students had 
caught up with those in the original treatment 
group. The authors point out that although it is 
encouraging that students could benefit from the 
tutoring during either semester of the first year, 
it is curious that students in the first treatment 
group were not able to build on the tools that 
they received from the extra tutoring and out- 
pace the original controls. They concluded that 
the question is outside the scope of their study, 
but they do point out that even though both 
groups made progress, at the end of their first 
year at school, even with 40 sessions of one-on- 
one tutoring, both groups were essentially only 
at a pre-primer level. 

305. Baker, Gersten, and Keating (2000). 



Schaefer and Jo (2005) studied a seven- 
week reading summer camp for 1st grad- 
ers. Exercises during reading time were 
designed to address specific skill sets; for 
certain exercises, students were matched 
according to their skill levels for paired 
and small group reading instruction. The 
study found large positive effects on de- 
coding and reading comprehension in fol- 
low-up test scores from September. 

Borman, Goetz, and Dowling (2008) ran- 
domly assigned exiting kindergarten stu- 
dents to participate in either a six-week 
summer program or a no-treatment con- 
trol group. The program found positive ef- 
fects on some literacy skills.^®® Class sizes 
were limited to 10, and the pacing of the 
morning literacy block was determined 
by student needs. College student interns, 
who worked with certified teachers, par- 
ticipated in weekly professional develop- 
ment workshops with the teachers and 
other experts. They also had a four-week 
training program on curricula/instruction, 
assessment, classroom management, par- 
ent involvement, and team building before 
the summer camp began. 

Inconclusive evidence for one-on-one 
tutoring in OST programs 

Two other small sample studies on after- 
school one-on-one or one-on-two programs 



306. Borman, Goetz, and Dowling (2008) had five 
measures of early literacy: letter naming, pho- 
neme segmentation, word list, developmental 
reading Instruction (DRA), and dictation. There 
were no significant differences found between the 
treatment and control groups on letter naming, 
phoneme segmentation, or dictation. After con- 
trolling for multiple comparisons following WWC 
guidelines, the average pre-to-post difference for 
word lists was significantly greater for the treat- 
ment group compared to the control group. Also, 
both word lists and DRA had estimated effect 
sizes greater than 0.25 using Hedges’s g (for tech- 
nical details on both computations, see http://ies. 
ed.gov/ncee/wwc/references/iDocViewer/Doc. 
aspx?docld=20&tocld=6). 
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found no effects but were judged to have 
lower levels of correspondence. McKin- 
ney (1995) conducted a randomized con- 
trolled trial on Leap Frog, an after-school 
program in which university students 
provided one-on-one tutoring and home- 
work assistance to 1st and 2nd graders, 
but since teachers from the school as- 
signed topics and assignments for the tu- 
tors to cover, this program could be only 
as adaptive as the teachers were. Udell 
(2003) used a quasi-experimental design 
to study the NDP program, which provided 
one-on-two reading tutoring to 2nd-grade 
students after school. Student assessment 
and progress monitoring were emphasized 
in the program, and tutors were trained to 
assess their students’ strengths and needs 
and to determine appropriate educational 
objectives and lessons based on those as- 
sessments. The tutors also were trained 
in effective instructional and motivational 
strategies, and they were provided with 
ongoing coaching from an experienced 
schoolteacher. However, this was a rela- 
tively short program consisting of fewer 
than 20 hours of literacy tutoring. 

Mixed evidence on individualizing 
instruction in OST programs 

The next three programs are larger stud- 
ies of two summer school programs (Teach 
Baltimore and Chicago Summer Bridge) and 
one after-school program (Enhanced Aca- 
demic Instruction). Only the Enhanced Aca- 
demic Instruction program was considered 
highly relevant to this recommendation. 

Borman and Dowling (2006) failed to find 
overall effects of the Teach Baltimore pro- 
gram on reading despite attempts to in- 
dividualize instruction, but they did find 
positive academic effects when compar- 
ing high attendees to students whom they 
judged more likely to have been high at- 
tendees (see discussion in this appendix 



307. McKinney (1995); Udell (2003). 



for recommendation 2). This could be 
interpreted as a somewhat cautionary 
note of support for some elements of the 
program. It is not possible to disentangle 
which components of the Teach Baltimore 
program are responsible for its success 
with high attendees. Whether this should 
be attributed to the adaptive nature of the 
instruction, the engaging elements of the 
instruction, the longevity of the program 
(available to students for three summers 
in a row after 1st, 2nd, and 3rd grades), 
interactions of various components, or 
something else entirely is not known. 

The Teach Baltimore summer program used 
a curriculum that was designed to be adap- 
tive to varying levels of achievement and 
provided instructors with materials they 
could use to supplement core instruction 
based on students’ progress. The program 
also provided three weeks of training to its 
instructors (undergraduate students) prior 
to the program and the maximum class size 
was held to eight students. 

Jacob and Lefgren (2005) studied Chicago’s 
Summer Bridge program and found effects 
in math and reading for 3rd graders but 
not for 6th graders. The program was de- 
signed to provide remedial instruction in 
small classes of around 16 students. In 
a qualitative study of twenty-six 6th grad- 
ers who participated in Summer Bridge, 
some students reported receiving indi- 
vidualized instruction, and a few reported 
frustration with the overall slow pace of 
the classes, and their resulting inability to 
learn according to their needs. Despite 
the relatively small class sizes compared 
to in-school instruction, the program was 
narrowly focused on pacing and keeping 



308. Borman and Dowling (2006). 

309. Roderick, Jacob, and Bryk (2004). 

310. Stone et al. (2005). Of the twenty-six 6th 
graders interviewed, 10 were classified as hav- 
ing positive experiences, 13 with neutral expe- 
riences, and 3 with negative experiences. Only 
the ones with overall negative experiences men- 
tioned frustration with the pacing. 
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up with a specific, highly structured, cen- 
trally dictated curriculum and may not 
actually have been adequately adaptive to 
student needs. 

Black, et al. (2008) studied an after-school 
program of Enhanced Academic Instruc- 
tion in math and reading (separate treat- 
ment groups) and found small positive 
effects for the math curriculum but not 
for the reading curriculum. Both curricula 
were designed to assess students before 
and after lessons, provide instruction 
in small groups (an average of nine stu- 
dents could be broken down into smaller 
groupings) separated by grade or skill, as 
well as time for students to work indepen- 
dently with self-monitoring and supervi- 
sion from an instructor. Instructors were 
provided with training, manuals, and mul- 
tiple sources of ongoing support to enable 
effective implementation of the program. 
However, the comparison group also re- 
ceived after-school programming with a 
low teacher-to-student ratio. Neverthe- 
less, a definite service contrast between 
the treatment group and the control group 
was the use of formative diagnostic testing 
to influence the direction of the instruc- 
tion, since after-school programs were 
specifically chosen to host the Enhanced 
Academic Instruction program only if they 
did not use diagnostic tests to guide their 
programming in their regular offerings. 

Cautionary evidence on 
individualizing instruction in 
out-of-school time programs 

This section discusses the 21st CCLC pro- 
grams and SES, neither of which has dem- 
onstrated positive academic effects. 



311. In the math study, the treatment group ratio 
was 1:9 compared to 1:11 in the regular after- 
school program. In the reading study, the treat- 
ment group ratio was 1:9 compared to 1:14 for 
the regular group. 



A large number of 21st CCLC programs 
were evaluated in the national study.^^^ 
As expected, the implementation of the 
program was not consistent across all the 
programs. Although an academic compo- 
nent was mandated by law, in practice, 
many programs put a strong emphasis on 
recreation because it was more attractive 
to students. Programs that did focus on 
academics often provided tutoring in small 
groups of 5 to 7 students or test prepa- 
ration classes in groups of 7 to 10. They 
also may have made an effort to monitor 
student progress in the program and at 
school and aligned instruction with input 
from the schoolteacher. Many programs 
required students to participate in the 
academic session before any recreational 
activities, but the academic session could 
have been as simple as unstructured time 
to complete homework with minimal fol- 
low-up from a staff member. 

The literature on SES, an intervention that 
is typically but not always provided one- 
on-one or in small groups, is not encourag- 
ing. Zimmer et al. (2007) found some posi- 
tive effects in a large national study that 
does not meet WWC standards because the 
treatment and comparison groups were 
not equivalent initially.^^^ Three quasi- 
experimental studies that met WWC stan- 
dards with reservations found no aca- 
demic effects; the authors of two of these 
studies suggest that the dosages of SES 
provided in the locales they studied (cities 
in Tennessee and in Louisville, Kentucky) 



312. U.S. Department of Education (2003); Dy- 
narski et al. (2004); James-Burdumy, Dynarski, 
and Deke (2007). 

313. U.S. Department of Education (2003). 

314. As mandated by law, if all students eligible 
for SES cannot receive services, schools must pri- 
oritize lower-performing students to receive ser- 
vices first. Zimmer et al. (2007), in their study of 
SES in nine large, urban districts, did find that at 
baseline, the eligible students who received SES 
had significantly lower achievement levels than 
their comparison group peers, eligible students 
who did not receive SES. 
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were insufficient. However, the dosages 
(30-40 extra hours a year) are comparable 
to at least one of the programs with posi- 
tive effects mentioned earlier, the Expe- 
rience Corps Book Buddies program. 
The authors also suggest that the lack of 
results could result from a lack of sensi- 
tivity of standardized state tests to small 
changes in achievement levels. 



Supplemental evidence from 
other sources 

Three studies were gleaned for illustra- 
tions of practices relevant for individual- 
izing instruction even though they did not 
pass WWC screens for this guide. One 
study looked at the Early Start to Emancipa- 
tion Preparation (ESTEP) tutoring program, 
which was organized at community colleges 
instead of by a school or district. The 
program incorporated frequent assessment 
of students, including diagnostics, weekly 
progress tests, and other periodic checks. 
It also emphasized instructing students at 



315. Tennessee — Ross et al. (2008); Louisville, 
Kentucky — Munoz, Potter, and Ross (2008). The 
third study, McKay et al. (2008), did not provide 
any information on dosage. 

316. The Experience Corps Book Buddies pro- 
gram in the Bronx, New York (Meier and Invernizzi 
2001) found positive effects of forty 45-minute 
tutoring sessions (30 hours total) over the course 
of a semester. The Howard Street Tutoring Pro- 
gram (Morris, Shaw, and Perney 1990) met with 
students twice a week for one hour. The program 
lasted from early October until late May; poten- 
tially, the tutees had as much as 54 hours of extra 
instructional time over the course of the school 
year (2 hours a week for 32 weeks but subtracting 
2 weeks for the winter holidays, 1 each for spring 
break and for the first and last weeks of the eight- 
month time span). Unfortunately, data on actual 
extra tutoring time are not provided. 

317. Tennessee — Ross et al. (2008); Louisville, 
Kentucky — Munoz, Potter, and Ross (2008). 

318. Courtney et al. (2008); New York City Board 
of Education (1991); Brown (2002). 

319. Courtney et al. (2008). This study was out 
of the scope for the practice guide because of 
age range. 



their own pace and trained tutors accord- 
ingly. The Project CARES (Career Aware- 
ness Resources for Exceptional Students) 
program also did diagnostic needs assess- 
ments of students and included student- 
paced instruction and computer-assisted 
adaptive teaching.^^o study of the Boys 
and Girls Club of Broward County program 
described the instructors grouping students 
by achievement levels whenever possible 

during academic instruction. 

Three programs with explicit staff devel- 
opment practices provided concrete ex- 
amples for implementation strategies even 
though the evaluations of these programs 
either did not pass WWC screens^^^ or did 
not meet WWC standards. Evaluations of 
Communities Organizing Resources to Ad- 
vance Learning (CORAL) have been purely 
qualitative or lacked a comparison group. ^^4 
The evaluations of the other two programs. 
Building Educated Leaders for Life (BELL) 
and Partnership Education Program (PEP), 
either did not measure initial achievement 
levels (BELL)^^^ or had treatment and com- 
parison groups that were not equivalent 



320. New York City Board of Education (1991). 
The study did not pass screens because it lacked 
a comparison group and seemed to be targeting 
high school students. 

321. Brown (2002). The study did not have a com- 
parison group nor was it school based. 

322. Arbreton et al. (2008); Sheldon and Hop- 
kins (2008). 

323. Chaplin and Capizzano (2006); Tucker 
(1995). 

324. Arbreton et al. (2008) and Sheldon and 
Hopkins (2008) did not pass WWC screens for 
this reason. 

325. Chaplin and Capizzano’s (2006) randomized 
controlled trial did not meet standards because 
it suffered from high attrition and had not mea- 
sured the initial achievement levels, so it could 
not demonstrate that the treatment and control 
groups were equivalent at baseline. Furthermore, 
the posttest achievement tests for students in the 
control group were administered on a different 
date than those for the treatment group and 
were, therefore, not directly comparable. 
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before the program began (PEP).326 Hence, 
neither study met WWC standards. 

In the BELL program, instructors were pro- 
vided with training and manuals to pre- 
pare them for teaching in the summer pro- 
gram. In PEP, there was a strong emphasis 
on observing after-school instruction of stu- 
dents and monitoring for quality, as well as 
coaching instructors in the proper instruc- 
tional practices. CORAL, an after-school 
literacy program for grades K-8, provided 
individualized training and coaching to its 
instructors by grouping them by experience 
and prior training and tailoring the training 
to the needs of the instructors. 

Recommendation 4. 

Provide engaging learning 
experiences 

Level of Evidence: Low 

The panel judged the level of evidence for 
this recommendation to be low. Studies of 
the types of activities covered in this rec- 
ommendation have demonstrated that they 
are effective in laboratory and school-day 
settings. However, the evidence for whether 
these practices are effective is mixed in 
the OST context. Although many programs 
identified making instruction engaging as a 
key program goal, very few demonstrated 
consistently positive effects, and none 
linked positive effects directly to the use of 
the strategies outlined in this recommenda- 
tion. The panel believes programs can be 
more successful if they implement engag- 
ing learning strategies that are integrated 
into the programs’ academic components. 



Summary of evidence 

Several studies, conducted outside of the 
OST arena, have examined the effectiveness 



326. Tucker (1995). 

327. Capizzano et al. (2007). 

328. Tucker (1995). 



of different strategies to increase student 
motivation, engagement, and academic 
success. In the OST context, however, 
the evidence supporting the use of engag- 
ing activities has been mixed (see Table 
D6). Eive programs documented practices 
highly aligned with those recommended 
by the panel; these either made a deliber- 
ate effort to integrate engaging practices 
with academic content or intentionally de- 
veloped relationships between students 
and OST staff with the objective of en- 
gaging students with school and learn- 
ing_33o Qf these, three programs demon- 
strated positive academic effects, and 
two showed mixed effects. Six other pro- 
grams had practices similar to the panel’s 
recommendations, but they either did not 
provide enough descriptive evidence to de- 
termine whether they were highly aligned 
with the panel’s recommendations or con- 
tained some evidence that the strategies 
were used in a way that was inconsistent 
with the panel’s recommendations. One 
of these showed positive effects, one 



329. Cordova and Lepper (1996); Anderson 
(1998); Guthrie et al. (1999); Guthrie, Wigfield, 
and VonSecker (2000); Battistich et al. (1997); 
Klem and Gonnell (2004); Helme and Glark (2001); 
Blumenfeld and Meece (1988); Gonnell and Well- 
born (1991). 

330. Early Risers — ^August et al. (2001); Developmen- 
tal Mentoring — Karcher, Davis, and Powell (2002); 
Enhanced Academic Instruction — Black et al. (2008); 
Teach Baltimore — Borman and Dowling (2006); Kin- 
dergARTen — Borman, Goetz, and Dowling (2008). 

331. Early Risers — August et al. (2001); Devel- 
opmental Mentoring — Karcher, Davis, and Pow- 
ell (2002); KindergARTen — Borman, Goetz, and 
Dowling (2008). 

332. Enhanced Academic Instruction — Black et 
al. (2008); Teach Baltimore — Borman and Dowl- 
ing (2006). 

333. Summer Reading Day Gamp — Schacter and 
Jo (2005); Chicago Summer Bridge — ^Jacob and 
Lefgren (2004); L.A.’s BEST — Goldschmidt, Huang, 
and Chinen (2007); YS-GARE — Bissell et al. (2002); 
21st GGLC — U.S. Department of Education (2003); 
NDP— Udell (2003). 

334. Summer Reading Day Camp — Schacter and 
Jo (2005). 



73 



APPENDIX D. TECHNICAL INFORMATION ON THE STUDIES 



Table D6. Studies of programs cited in recommendation 4 that met WWC standards 
with or without reservations 



Program and Study Details 


Strategies 


Brief Citation 


Program 

Type 


Grades 

Studied^* 


Level of 
Relevance*’ 


Integrated 

Engaging 

Components 

with 

Academic 

Components 


Examples of 
Activities 
Used to Make 
Learning 
Relevant 


Examples 
of Active 
Learning 
Experiences 


Fostered 
Positive 
Adult- 
Student 
Relationships 
to Increase 
Engagement 
in Learning 


Positive Academic Effects 


KindergARTen, Borman, 
Goetz, & Dowling (2008) 


Summer 


K 


VV 


Yes 


Themes, field 
trips 


Collaborative 

learning 




Summer Reading Day 
Camp, Schacter & Jo 
(2005) 


Summer 


1 


V 




Personalize 
learning 
to student 
experiences 


Hands-on, 

exploratory 

learning 




Early Risers, August 
et al. (2001) 


Summer 
and school- 
year 

mentoring 


K-2 


VV 






Participation, 

role-play 


Teachers, 

parents 


Developmental 
Mentoring, Karcher, 
Davis, & Powell (2002) 


Summer 
and school- 
year 

mentoring 


5 


VV 




Themes, 
personalize 
learning 
to student 
experiences 




Teachers, 

parents 


Mixed Academic Effects 


Teach Baltimore, Borman 
& Dowling (2006) 


Summer 


K-2 


VV 


Yes 


Field trips 


Hands-on, 

games 




Chicago Summer Bridge, 
Jacob & Lefgren (2004) 


Summer 


3, 6 


V 






Interactive 

questioning, 

collaborative 

learning 




Enhanced Academic 
Instruction, Black et al. 
(2008) 


After 

school 


2-5 


VV 


Yes 


Personalize 
learning 
to student 
experience 


Hands-on, 

games, 

computers 




No Detectable Academic Effects 


NDP, Udell (2003) 


After- 

school 

tutoring 


2 


V 








Program staff 


L.A.’s BEST, 
Goldschmidt, Huang, 
& Chinen (2007) 


After 

school 


K-6 


V 






Recreational 
activities — 
non-linked 




21st CCLC, U.S. Depart- 
ment of Education (2003); 
Dynarski et al. (2004); 
James-Burdumy et al. 
(2005) 


After 

school 


K-8 


V 


No 




Engaging 

programming 




YS-CARE, Bissell et al. 
(2002) 


After 

school 


1-5 


V 




Choice 


Enrichment 

activities 





a. For summer programs, refers to the grade participating students exited prior to the commencement of the summer program. 

b. indicates reievance of recommendation to program operation. VV” indicates high relevance; V” indicates lower relevance. 
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showed mixed effects, and four showed 
no effects. Although the evidence is 
somewhat mixed, on average, those with 
positive or mixed effects seemed to be 
more closely aligned with the strategies 
recommended by the panel. 

Evidence from non-OST studies 

Engagement is a multidimensional con- 
cept that can refer to behavior (ranging 
from following classroom rules to ac- 
tively participating in class and the school 
community),^^^ emotions (affectations re- 
lated to learning including excitement and 
boredom or identification with school),^^® 
and cognition (effort and motivation to 
learn). Many studies have shown that 
engagement is correlated with improved 
academic outcomes for students^^° and 
that behavioral disengagement is associ- 
ated with poor academic performance. 
Activities that include choice and coopera- 
tion between staff and students have been 
linked theoretically to student engagement 
and connection to school. Likewise, 
teacher support and caring are associated 
with increased engagement. There is 
some evidence that autonomy and choice 



335. Chicago Summer Bridge — ^Jacob and Lefgren 
(2004). 

336. L.A.’s BEST — Goldschmidt, Huang, and 
Chinen (2007); YS-CARE— Bissell et al. (2002); 
21st CCLC — U.S. Department of Education (2003); 
NDP— Udell (2003). 

337. See Finn (1993); Finn, Pannozzo, and Voelk 
(1995) for more on behavioral engagement. 

338. See Finn (1989), Connell and Wellborn (1991), 
and Skinner and Belmont (1993) for more on emo- 
tional engagement. 

339. See Connell and Wellborn (1991) and Newmann 
et al. (1992) for more on cognitive engagement. 

340. Connell, Spencer, and Aber (1994); Marks 
(2000); Wellborn and Connell (1990); Connell and 
Wellborn (1991). 

341. Finn, Pannozzo, and Voelkl (1995); Finn and 
Rock (1997). 

342. Newmann et al. (1992). 

343. Battistich et al. (1997); Klem and Connell 
(2004). 



encourage students to persist and engage 
more with learning activities. Other re- 
searchers have argued that collaboration, 
“opportunities for fun,” “novel tasks that 
have personal meaning,” hands-on activi- 
ties, and tasks that build on individual tal- 
ents can increase engagement. 

In a study using student records and sur- 
vey data, Klem and Connell (2004) looked 
at how student engagement and levels of 
teacher support for students correlated 
with future academic achievement in six 
elementary and three middle schools in an 
urban district. They found that more highly 
engaged students had higher achievement 
than did those who were less engaged 
(based on self and teacher reports of en- 
gagement). Additionally, the authors found 
that when students perceived their teach- 
ers to be supportive, the students tended 
to be more engaged in school. 

The panel also considered two specific ex- 
perimental studies to supplement the evi- 
dence on this recommendation. The first 
examined adding personalized components 
(such as the student’s name or birthday), 
choice, and embellishments designed to 
pique student interest in a computer game 
designed to teach order of operations. When 
the game included these components, stu- 
dents demonstrated increased engagement 
in and learning from the tasks. In the 
second study, students in six classrooms 
at two schools were randomly assigned to 
either a control condition that received the 
traditional science textbook or one of three 
treatment conditions. One treatment group 
received interesting texts about the animals 



344. Turner (1995); Perry (1998). 

345. See, for example, Newmann (1991); Helme and 
Clark (2001); Blumenfeld and Meece (1988); Guthrie, 
Wigfield, and VonSecker (2000); Connell and Well- 
born (1991). Most of the information in this para- 
graph is drawn from Fredricks, Blumenfeld, and 
Paris (2004), which reviews the available research 
on student engagement in school environments. 

346. Klem and Connell (2004). 

347. Cordova and Lepper (1996). 
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students were studying, another was given 
time to observe animals, and the third re- 
ceived both the interesting texts and the 
observation opportunities. All groups were 
instructed by the researcher and rotated to 
work with her in small groups in a separate 
classroom over the course of a one-week 
intervention. The researcher reported 
increased conceptual knowledge among 
students receiving the interesting texts and 
the combination of observation and inter- 
esting texts, relative to the control group, 
but not among the students in the obser- 
vation alone group. This study provides 
some evidence for the use of engaging in- 
structional materials and opportunities for 
real-world interactions, but it is limited by a 
small sample, short duration, and possible 
contamination of the groups when students 
returned to the school-day classroom and 
had opportunities to discuss their experi- 
ences in small groups. 

Finally, two correlational studies provide 
additional support for the incorporation 
of science observation and engaging in- 
struction into classroom learning. Con- 
cept-Oriented Reading Instruction (CORl) 
integrates science content and engaging in- 
struction (including real-world, hands-on 
activities and collaboration) into reading 
instruction. Guthrie et al. (1999) found 
that students instructed using these meth- 
ods showed greater concept learning and 
comprehension gains than did students in 
traditional instruction classrooms. In an- 
other study, the same strategies produced 
gains in motivation and engagement for 
participating students. 



348. Anderson (1998) acknowledges the possibil- 
ity of experimenter effects unintentionally influ- 
encing the results of the study. 

349. Ibid. 

350. Guthrie et al. (1999). 

351. Guthrie, Wigfield, and VonSecker (2000). 



Positive OST evidence for 
engaging instruction 

Four programs with engaging instruction 
showed positive effects in randomized con- 
trolled trials. 352 Three of the four evalua- 
tions discussed practices that were closely 
related to this recommendation, 353 and 
the fourth study did not contain enough 
detail to judge the level of integration of 
engaging practices in the academic in- 
struction. 354 Two of the closely related pro- 
grams that showed positive effects worked 
intentionally to engage and connect stu- 
dents with learning by developing positive 
adult and peer relationship through men- 
toring and outreach to parents, as recom- 
mended by the panel in the third action 

step for this recommendation. 355 

Early Risers is a two-year tutoring and 
mentoring program that recruits children 
with early onset aggressive behavior at 
the end of kindergarten. 356 The study was 
conducted in two locations each with 10 
schools randomly assigned to treatment 
and control groups and found consistently 
positive academic effects. The program 
includes a six-week summer component 
and additional support for students, fami- 
lies, and teachers during the school year. 
Students were paired with peer mentors 
to collaborate daily on specific tasks de- 
signed to develop social skills but which 
also might have contributed to the devel- 
opment of relationships and positive en- 
gagement within a school environment 



352. August et al. (2001) evaluated a summer pro- 
gram with year-round mentoring; Karcher, Davis, 
and Powell (2002) and Karcher (2005) studied 
a developmental mentoring program; Borman, 
Goetz, and Dowling (2008) and Schacter and Jo 
(2005) evaluated different summer programs. 

353. August et al. (2001); Karcher, Davis, and Pow- 
ell (2002); Borman, Goetz, and Dowling (2008). 

354. Schacter and Jo (2005). 

355. August et al. (2001); Karcher, Davis, and 
Powell (2002); Karcher (2005). 

356. August et al. (2001). 
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that the panel believes will promote aca- 
demic success. The program also provides 
teachers and parents with strategies to 
manage children’s behavior, ultimately 
intending to develop stronger relation- 
ships with other adults. Fidelity-monitor- 
ing checklists measured whether summer 
staff members encouraged student partici- 
pation or used role-play, both components 
of the engaging instruction recommended 
by the panel. 

Two studies looked at the effectiveness 
of a Saturday and summer school pro- 
gram that used developmental mentoring 
to increase connectedness and academic 
achievement. The first study demon- 
strated small positive academic effects, 
and the second study showed that the pro- 
gram increased students’ connectedness 
on multiple dimensions. In Karcher et al. 
(2002), 30 average or above average achiev- 
ing 5th-grade students from three schools 
with high dropout rates were randomly 
assigned to either a no-treatment com- 
parison group or to the program, which 
consisted of monthly Saturday program- 
ming and a two-week summer session 
with high school mentors. The program 
included academic enrichment and social 
connectedness components. Participating 
students integrated math, science, writing, 
and computer skills into a final project 
over the summer. Students also completed 
writing projects that incorporated social 
and cultural experiences, future careers, 
and negotiation and cooperation. The pro- 
gram encouraged parents to attend pro- 
gramming and field trips. A second study 
of a similar program produced greater con- 
nectedness to school and learning but did 
not measure whether there were academic 
achievement gains. 



357. Ibid. 

358. Karcher, Davis, and Powell (2002); Karcher 
(2005). 

359. Karcher (2005). 



The other closely related program docu- 
mented an intentional effort to integrate 
field trips and activities with the academic 
goals of the program. Borman, Goetz, 
and Dowling (2008) randomly assigned ex- 
iting kindergarten students to participate 
in either a six-week summer program or a 
no-treatment control group. The program 
found positive effects on some literacy 
skills. The program, intended to “build 
a love of learning,” included weekly field 
trips and theme-based afternoon activities 
designed to build background and con- 
tent knowledge for the literacy instruction 
conducted each morning. poetry and 
songs, thematic learning, and collabora- 
tive learning activities were regular pro- 
gram components. 

Finally, Schacter and Jo (2005) studied a 
seven-week summer reading day camp in 
which students exiting 1st grade in three 
high-poverty schools with large minor- 
ity populations in south Los Angeles were 
instructed using writing activities that in- 
corporated everyday life experiences and 
creative writing opportunities.^®^ Play and 
exploration were encouraged. Students were 
randomly assigned either to participate in 
the camp (maximum enrollment was lim- 
ited to 72 students because of the amount 
of funding) or to serve as a control group. 
The study found large positive effects on 



360. Borman, Goetz, and Dowling (2008). 

361. Borman, Goetz, and Dowling (2008) had five 
measures of early literacy: letter naming, phoneme 
segmentation, word list, developmental reading in- 
struction (DRA), and dictation. There were no signifi- 
cant differences found between the treatment and 
control groups on letter naming, phoneme segmen- 
tation, or dictation. After controlling for multiple 
comparisons following WWG guidelines, the aver- 
age pre-to-post difference for word lists was signif- 
icantly greater for the treatment group compared 
to the control group. Also, both word lists and DRA 
had estimated effect sizes greater than 0.25 using 
Hedges’s g (for technical details on both computa- 
tions, see http://ies.ed.gov/ncee/wwc/references/ 
iDocViewer/Doc.aspx?docId=20&tocId=6). 

362. Borman, Goetz, and Dowling (2008), p. 2. 

363. Schacter and Jo (2005). 
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decoding and reading comprehension in 
follow-up test scores from September. 

Mixed support for engaging 
instruction in OST 

Three large studies provide mixed sup- 
port for the use of engaging instructional 
practices in In two, engaging in- 

structional techniques were intentionally 
integrated with the academic content of 
the program. In the third, supplemental 
information from qualitative interviews 
with students suggests that engaging in- 
struction in the OST offering might be 
more common than during school, but this 
was not represented as a core objective of 
the program.^®® 

Chicago Summer Bridge showed positive 
effects on student achievement among 
the districts’ lowest-performing Srd-grade 
students, but not among 6th graders. 
Participating students reported improved 
relationships with teachers and indicated 
that they received more encouragement to 
achieve from summer instructors than from 
their teachers during the school year. Stu- 
dent interviews also suggested that some 
instructors were able to keep students en- 
gaged by using interactive questioning to 
gauge student understanding and collabor- 
ative pairing to support the needs of both 
struggling and more advanced students; 
however, none of these components was 
presented as a major or consistent compo- 
nent of Chicago Summer Bridge.^®® 

A randomized controlled trial of the Teach 
Baltimore program showed no positive ef- 
fects on reading, but a quasi-experimental 



364. Borman and Dowling (2006); Black et al. 
(2008); Jacob and Lefgren (2004). 

365. Borman and Dowling (2006); Black et al. 
(2008). 

366. Stone et al. (2005). 

367. Jacob and Lefgren (2004). 

368. Stone et al. (2005); Roderick et al. (2004). 



component showed positive significant ef- 
fects for students who attended the summer 
program more frequently. In the program, 
students participated in hands-on math 
and science learning projects and education 
games in the afternoon and also learned 
new skills and knowledge through field trips 
to museums, which were integrated with 
classroom activities. Students also incorpo- 
rated experiences beyond the classroom and 
neighborhood into the learning.^®^ 

A study of Enhanced Academic Instruc- 
tion in After-School Programs randomly 
assigned students to receive either a struc- 
tured enhanced curriculum or a regular 
after-school day offering. The instructional 
materials and program were intentionally 
designed to be engaging and related to 
academic standards, which led to the in- 
clusion of games, hands-on activities, and 
computers, and program staff members 
were evaluated on instructional techniques, 
including whether they made efforts to link 
academic concepts to student experiences. 
The program demonstrated small math 
effects; however, a parallel intervention 
for reading, which included many similar 
features and also was intended to engage 
students with academic content, showed no 
effects. This suggests caution in interpret- 
ing the study findings as evidence for the 
effectiveness of engaging instruction. 



Inconclusive evidence for engaging 
instruction in OST 

Four programs at least partially imple- 
mented the panel’s recommendations and 
showed no effects, but studies of each ei- 
ther provide evidence that engaging activi- 
ties were not clearly linked to academic in- 
struction or do not contain enough detail 
for the panel to judge their alignment with 



369. Borman and Dowling (2006). 

370. Black et al. (2008). 



78) 



APPENDIX D. TECHNICAL INFORMATION ON THE STUDIES 



the recommended practices. A quasi- 
experimental study of the NDP program, 
which provided one-on-two or one-on-one 
reading tutoring from undergraduate tu- 
tors to 2nd-grade students after school, 
found no effects compared to a compari- 
son group that received small group ac- 
tivities not related to literacy.^^^ jhg study 
sample was small (27 students in the treat- 
ment and control groups combined), and 
the treatment duration was short (twice a 
week for 10 weeks). In the program, tutors 
used opportunities for creative expression 
to supplement reading instruction. Based 
on qualitative reflections from tutors, the 
researcher observed that as the tutors de- 
veloped relationships with their students, 
they were better able to engage them in the 
instruction. However, these relationships 
were brief and not described as a goal of 
the program. There is no information in 
the study to determine whether fostering 
engagement with school or learning was 
an intentional or widespread component 
of the program. 

Like NDP, studies of L.A.’s BEST and YS- 
CARE do not provide enough detail to de- 
termine the level of integration between 
academic and engaging components of the 
program. A quasi-experimental study 
of L.A.’s BEST found no consistent posi- 
tive academic effects. Although some 
engaging activities were not strongly tied 
to academic programming (recreational 
activities included cooking, holiday activi- 
ties and crafts, for example^^®), promot- 
ing student engagement was identified as 
a program goal in supplementary studies 



371. Udell (2003); Bissell et al. (2002); Gold- 
schmidt, Huang, and Chinen (2007); U.S. Depart- 
ment of Education (2003). 

372. Udell (2003). 

373. Ibid. 

374. Goldschmidt, Huang, and Chinen (2007); 
Bissell et al. (2002). 

375. Goldschmidt, Huang, and Chinen (2007). 

376. Ibid. 



of L.A.’s BEST,^^^ and academic activities 
were intended to promote a “love of learn- 
ing” and build relationships between stu- 
dents and staff.^^® Reports on L.A.’s BEST 
do not contain enough details about the 
actual implementation to judge the degree 
to which engaging activities were explic- 
itly linked to academic instructions. 
Similarly, an evaluation of the YS-CARE 
found positive but nonsignificant findings 
in the first year.^®° The program included 
diverse enrichment programs intended to 
broaden students’ knowledge base, among 
other objectives, and the enrichment staff 
traveled from site to site, but the specif- 
ics of these practices were not outlined in 
sufficient detail to determine the degree 
to which engaging practices were imple- 
mented in the program. 

finally, an evaluation of the 21st CCLC 
after-school program showed no consis- 
tent positive academic effects among el- 
ementary or middle school students but 
contains details that suggest that engaging 
practices were not fully integrated with the 
academic components of the programs. 
Participating middle school teachers re- 
ported improved relationships with stu- 
dents. Middle school site observations by 
the evaluation team revealed that student 
engagement was often higher during recre- 
ational and cultural activities, which were 
often intentionally disconnected from aca- 
demic components, than during academic 
classes. In most programs, students were 
allowed to select activities to increase pro- 
gram appeal, but choice often was limited 
to the recreation and cultural components. 
Although all programs offered some sort 
of academic assistance, it often was lim- 
ited to homework help sessions during 
which students worked independently 



377. Huang et al. (2007). 

378. Goldschmidt, Huang, and Chinen (2007), p. 9. 

379. Huang et al. (2007); Goldschmidt, Huang, 
and Chinen (2007). 

380. Bissell et al. (2002). 
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with little direct support from instructors. 
In elementary school, homework sessions 
were similarly structured and sometimes 
chaotic. In short, although many programs 
strove to provide engaging experiences for 
participants, these experiences often were 
disconnected from the academic compo- 
nents of the centers. 



Supplemental evidence 
from other sources 

Other studies provided examples of engag- 
ing practices, but they either did not meet 
the OST protocol for inclusion or failed 
to meet WWC standards with or without 
reservations. A number of meta-analyses 
have reviewed the effectiveness and char- 
acteristics of different OST programs. 
One found that programs that used so- 
cial and real-world examples had a posi- 
tive impact on math.^®^ Another reviewed 
73 reports with control groups on after- 
school programs that sought to improve 
personal and social skills. The authors 
found the strongest predictor of programs 
with increased academic achievement was 
that the program had an academic com- 
ponent.^®^ The authors also found that 
programs that were active or included op- 
portunities such as role-play and hands-on 
practice were among those that produced 
changes in student outcomes. 

In Project Adelante, a summer program 
and Saturday academy for recent Latino 
immigrants, promoting positive cultural 
identity was incorporated across courses 
and enrichment activities. All program- 
ming was aligned with a yearly theme. 
Students worked with mentors and in- 
structors from similar backgrounds who 
served as role models of successful Latino 



382. U.S. Department of Education (2003). 

383. Meta-analyses are ineligible for review 
under current WWC standards. 

384. Lauer et al. (2004). 

385. Durlak et al. (2007). 



professionals to improve student aspira- 
tions and expose them to real-world ca- 
reers. The program coordinated parent 
nights that included staff from the stu- 
dents’ schools, as well as meetings with 
parents to help them address concerns 
about the situations their children faced in 
school and to help them effectively man- 
age conflicts that emerged as their chil- 
dren adapted to a new culture.®®® 

The BELL program includes considerable 
academic instruction, described by the 
study authors as “culturally relevant”®®^ 
and arranged around specific themes, field 
trips, and guest speakers to supplement 
academic content.®®® Members of the panel 
reported observing particularly dynamic 
instructional settings. 

The TASC program showed consistently 
positive effects for math, but not for read- 
ing.®®® Roughly three-quarters of the site 
coordinators reported using a theme to 
link different program activities across 
years 2 to 4 of the project and that students 
participated in group work.®®® Although 
homework help was reported as the activ- 
ity with the most intense emphasis across 
TASC programs, recreational reading, math 



386. Center for Applied Linguistics (1994). This 
study failed to pass WWC screens because it did 
not contain a comparison group. 

387. Chaplin and Capizzano (2006), p. 5. 

388. Chaplin and Capizzano (2006). This ran- 
domized controlled trial did not meet standards 
because it suffered from high attrition and had 
not measured the initial achievement levels and 
could not demonstrate that the treatment and 
control groups were equivalent initially. Further- 
more, the posttest achievement tests for students 
in the control group were administered on a dif- 
ferent date than those for the treatment group 
and were, therefore, not directly comparable. 

389. Reisner et al. (2004). This study found posi- 
tive and significant math gains for participants, 
but no effects for language arts. Because baseline 
equivalence of the treatment and control groups 
was not reported, WWC reviewers were unable to 
assign a rating to the study. 

390. Ibid. 
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games, and word games were among the 
top 5 out of 12 activities measured. 

A qualitative study of the Philadelphia Bea- 
con Centers combined surveys of students 
and staff with observations of program of- 
ferings to identify practices that engaged 
students and observers. The study found 
the use of cooperative learning and choice 
was associated with student enjoyment 
and desire to participate in activities. 

A study of the Voyager TimeWarp model, 
implemented in 13 middle and elemen- 
tary schools for low-achieving students, 
includes collaborative learning as a key 
component. In another small program, 
professional women with careers in the 
physical sciences mentored minority girls 
in classroom settings designed to give 
them hands-on experience with science 
and provide them with role models in 
the field. Related field trips showed the 
classroom lessons in action. 

When implementation data suggested that 
the CORAL program (designed primarily 
for elementary students in low-performing 
schools in California) was poorly imple- 
mented on average and consisted mostly of 
homework help and enrichment activities 
that were disconnected from academic con- 
tent, the program tried to improve program 
quality and student engagement. As the 
program implemented changes, specifically 
enhancing the academic component of the 
program, most CORAL students reported 



391. Ibid. 

392. Grossman, Campbell, and Raley (2007). This 
study failed to pass WWC screens because it did 
not attempt to measure program impacts on aca- 
demic achievement. 

393. Roberts and Nowakowski (2004). This study 
failed to pass WWC screens because it did not 
have a comparison group. 

394. Ferreira (2001). This study was not eligible 
for WWC review because it did not have a com- 
parison group. 

395. Arbreton et al. (2008). This study was not 
eligible for review because it did not include a 
comparison group. 



having adults they could talk to and who 
cared about them, and that they liked lit- 
eracy activities at the centers. Teachers 
reported efforts to make literacy activi- 
ties fun, including making connections 
between the stories they read and the stu- 
dents’ real-life experiences and engaging 
the students in conversations about the 
material. In some locations, enrichment 
staff rotated from site to site.^®® 

Recommendation 5. 

Assess program performance 
and use the results to improve 
the quality of the program 

Level of evidence: Low 

The panel judged the level of evidence 
supporting this recommendation to be low. 
Although the panel believes that monitor- 
ing and improving performance are impor- 
tant to ensure that the program is carrying 
out its intended objectives and adapting to 
changing needs and feedback, no direct 
evidence suggests that monitoring leads 
to increased academic achievement in OST 
programs. More research would be needed 
to isolate whether it is the monitoring itself 
or other components that lead to positive 
academic effects in those programs that 
do some form of monitoring. 



Summary of evidence 

OST programs could consider using dif- 
ferent kinds of assessments. Seven pro- 
grams relevant to this recommendation 
had studies that met WWC standards with 
or without reservations (see Table 
Of these seven, five included elements of 



396. Ibid. 

397. Early Risers — August et al. (2001); CHP — 
Langberg et al. (2006); Teach Baltimore — Borman 
and Dowling (2006); Chicago Summer Bridge — 
Jacob and Lefgren (2004); Enhanced Academic 
Instruction — Black et al. (2008); L.A.’s BEST — 
Goldschmidt, Huang, and Chinen (2007); YS- 
CARE— Bissell et al. (2002). 
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Table D7. Studies of programs cited in recommendation 5 that met WWC standards 
with or without reservations 



Program and Study Details 


Brief Description of Strategies Used 


Brief Citation 


Program 

Type 


Grades 

Studied^ 


Strategies Used for Evaluating Program Performance 


Positive Academic Effects 


Early Risers, August et al. 
(2001) 


Summer and 
school-year 
mentoring 


K-2 


Fidelity monitorine: Fidelity technicians observed instruction for 
content presentation, style of delivery, and staff comportment through 
unannounced visits. 

Checklists were used to assess the alignment of staff behavior along 
three dimensions: (a) elicits participation and shows enthusiasm, 

(b) uses skills training or reinforces skill use, and (c) uses behavior 
management techniques. 

Tracking student oarticination: Staff recorded attendance and material 
covered in logbooks, and level of engagement and homework completion 
in rating forms. 


CHP, Langberg et al. 
(2006) 


After 

school 


6-7 


Fidelity monitoring: Grouo leaders keot a folder for each child in which 
they recorded progress and behavior each time they met with a child. 

Staff supervisors reviewed the folders and identified problems during 
weekly supervision sessions. 


Mixed Academic Effects 


Teach Baltimore, Borman 
& Dowling (2006) 


Summer 


K-2 


Fidelitv monitoring: Instruction was observed to make sure it was being 
delivered as prescribed. 

Tracking student oarticipation: Student attendance also was monitored. 


Chicago Summer Bridge, 
Jacob & Lefgren (2004) 


Summer 


3, 6 


Fidelitv monitoring: Monitors visit the classroom regularly to observe 
instructors and make sure that they are teaching within the curriculum 
and pace guidelines. 


Enhanced Academic 
Instruction, Black et al. 
(2008) 


After 

school 


2-5 


Fidelitv monitoring: Local district coordinators were resnonsible for 
observing instruction. Representatives from the curriculum publishers 
also visited sites twice a year to observe instruction. 


No Detectable Academic Effects 


L.A.’s BEST, Goldschmidt, 
Huang, & Chinen (2007) 


After 

school 


K-6 


Fidelitv monitoring: The program’s office of evaluation has an internal 
evaluation team that regularly meets with field staff and administrators 
to share thoughts on program operation. 

Ongoing external evaluations: External evaluations are performed to 
measure the effect of the program on short- and long-term outcomes. 
Site-specific results are shared with site coordinators and used to evaluate 
individual site performance. 

Surveys of kev stakeholders: Evaluations gather feedback from program 
staff, school-day teachers, parents, and students. 


YS-CARE, Bissell et al. 
(2002) 


After 

school 


1-5 


Surveys of kev stakeholders: Evaluation surveys were administered 
to students, parents, school-day teachers, and YS-CARE staff to gauge 
satisfaction with the program. 



a. For summer programs, refers to the grade participating students exited prior to the commencement of the summer program. 
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fidelity monitoring in order to ensure that 
program implementation followed the pro- 
gram model, one used fidelity monitor- 
ing and ongoing external evaluations, 
and one program used surveys of key 
stakeholders to gauge satisfaction with 
the program.^°° Of the six that used fidel- 
ity monitoring, all were considered to be 
doing so in a way that was largely consis- 
tent with the panel’s recommendation, and 
five had either positive^°^ or partially posi- 
tive effects on academic achievement.^®^ 
The program that also used external evalu- 
ators and the program with stakeholder 
surveys were both inconclusive with small 
nonsignificant effects.^®^ Only the program 
that used just stakeholder surveys was 
considered to have a lower level of consis- 
tency with this recommendation because 
it was unclear how it used the surveys to 
improve the program. 

Two studies that monitored program 
implementation and had positive 
academic effects 

One randomized controlled trial of a sum- 
mer tutoring and school-year mentoring 
program (Early Risers) for exiting kinder- 
garten students with early onset aggres- 
sive behavior showed positive significant 
effects on academic competence.'^®^ Pro- 



398. Early Risers — August et al. (2001); CHP — 
Langberg et al. (2006); Teach Baltimore — Borman 
and Dowling (2006); Chicago Summer Bridge — 
Jacob and Lefgren (2004); Enhanced Academic 
Instruction — Black et al. (2008). 

399. E.A.'s BEST — Goldschmidt, Huang, and 
Chinen (2007). 

400. YS-CARE— Bissell et al. (2002). 

401. Early Risers — August et al. (2001); CHP — 
Eangberg et al. (2006). 

402. Teach Baltimore — Borman and Dowling 
(2006); Chicago Summer Bridge — ^Jacob and Eef- 
gren (2004); Enhanced Academic Instruction — 
Black et al. (2008). 

403. E.A.’s BEST — Goldschmidt, Huang, and 
Chinen (2007); YS-CARE— Bissel et al. (2002). 

404. August et al. (2001). 



gram monitoring was structured to ensure 
continuing fidelity with the model and 
included instructor logs of student atten- 
dance, their progress toward instructional 
objectives, and any changes in the imple- 
mentation of the program curriculum. Mon- 
itors conducted periodic unscheduled visits 
to observe staff compliance with program 
components, including checks on student 
engagement. Monitors used checklists to 
evaluate staff compliance with program 
standards. August et al. (2001) reported 
that the monitoring results indicated that 
the Early Risers program was implemented 
as intended.^®^ 

Langberg et al. (2006) also used a ran- 
domized controlled trial to study CHP, 
which provided academic remediation 
and study skills training in small groups 
after school for 6th and 7th graders with 
a combination of learning and behavior 
problems. The program showed no statis- 
tically significant effects on teacher rat- 
ings of academic progress because of the 
small sample size, but the effect size was 
0.45, large enough to be considered posi- 
tive evidence by WWC guidelines.^®® To 
monitor fidelity to the CHP model, group 
leaders were required to keep a folder on 
each CHP participant. The group leaders 
recorded the percentages of binder, book 
bag, and locker organization in the folder 
each time they met with a student. These 
folders were reviewed by the staff su- 
pervisors, and problems were addressed 
during weekly supervision. Also, staff su- 
pervisors were onsite every program day 
and observed counselors leading groups. 
Supervisors then addressed any program- 
adherence issues with counselors during 
the weekly group supervision.^®^ 



405. Ibid. 

406. Langberg et al. (2006). 

407. Ibid. 
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Mixed evidence for monitoring program 
implementation 

Three programs had mixed academic ef- 
fects but contained elements of fidelity 
monitoring. Whether they found positive 
academic effects depended on the analy- 
sis methods used and subgroups studied 
(Teach Baltimore), the student grade 
level (Chicago Summer Bridge), or the 
subject content area (Enhanced Academic 
Instruction in After-School Programs).'*^^ 
In the Teach Baltimore program, monitor- 
ing of instruction and student attendance 
was designed to ensure fidelity to the pro- 
gram model.^“ In Chicago Summer Bridge, 
monitors provided frequent oversight to 
ensure that instructors were implementing 
the desired curriculum and pacing, as well 
as to provide technical support.^^^ in the 
Enhanced Academic Instruction in After- 
School Math and Reading programs, bian- 
nual visits were conducted by curriculum 
publishers and independent evaluators to 
monitor instruction and pacing, as well as 
program implementation, and to provide 
feedback and follow-up training. Weekly 
phone calls with district coordinators were 
used to problem-solve challenges associ- 
ated with implementation of the model and 
to pass on implementation concerns that 



408. Borman and Dowling (2006) found no ef- 
fects on reading in an intent-to-treat randomized 
controlled treatment design that included all stu- 
dents but did find significant positive effects on 
reading using a quasi-experimental design that 
looked at the effects of the program only on the 
high attendees. See this appendix for recommen- 
dation 1 for more detail. 

409. Jacob and Lefgren (2004) found significant 
positive effects in both math and reading for 3rd 
graders but not for 6th graders. 

410. Black et al. (2008) found that their math 
curriculum had small but significant positive ef- 
fects on student achievement for 2nd through 5th 
graders, but their reading curriculum did not. 

411. Borman and Dowling (2006). 

412. Stone et al. (2005). Summer Bridge showed 
positive achievement effects for 3rd graders but 
not for 6th graders in Jacob and Lefgren (2004). 



had been raised in the weekly meetings 
with the OST instructors.^^^ 



Inconclusive evidence for fidelity or 
other types of monitoring 

Two programs, L.A.’s BEST and YS-CARE, 
were evaluated by quasi-experimental 
studies that found positive but small 
and nonsignificant effects on academic 
achievement.^^"* L.A.’s BEST internal evalu- 
ators regularly monitored staff to gather 
and share information on best practices. 
The program also commissioned external 
evaluators to gather feedback from pro- 
gram stakeholders and measure program 
effects. The data from these evaluations 
were shared with site coordinators to in- 
form program improvements and moni- 
tor site performance.^*^ The YS-CARE pro- 
gram administered surveys to program 
stakeholders (students, parents, in-school 
teachers, and program staff) to gauge their 
impressions of the program."**® 



Supplemental evidence 
from other sources 

Eor this recommendation, the panel sup- 
plemented the studies that met WWC stan- 
dards with sources that discussed for- 
mative and summative evaluations of 
OST programs and program quality as- 
sessment. A meta-analysis of summer 
programs provides some evidence that 
programs that conduct implementation 
monitoring may have larger effects com- 
pared to programs that do not.**^ Although 



413. Black et al. (2008). 

414. Goldschmidt, Huang, and Chinen (2007); Bis- 
sell et al. (2002). Neither study reported sufficient 
detail for effect sizes to be computed. 

415. Goldschmidt, Huang, and Chinen (2007). 

416. Bissell et al. (2002). 

417. Cooper et al. (2000). This meta-analysis 
showed positive effects for fidelity monitoring 
using fixed effects analysis, but not random 
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this finding is not fully generalizable, it 
provides some support for this recommen- 
dation, which the panel believes should be 
an important component of a successful 
OST program. 

A qualitative study of the CORAL program 
provided a few examples of how to carry 
out implementation monitoring. The 
program created tools for all observers of 
program instruction to use in rating the 
quality of teaching. The tools were used 
not only as a basis for providing targeted 
feedback for instructors, but also as a 
mechanism for measuring improvement 
over time in instruction and other aspects 
of the program. 



effects. Therefore, the authors caution that their 
findings should not be generalized beyond pro- 
grams similar to those included in the meta-anal- 
ysis. Furthermore, they point out, “These associa- 
tions may be due to the impact that surveillance 
can have on the rigor with which programs are de- 
livered. Or, they may simply indicate that the extra 
care that monitoring of programs implies is asso- 
ciated with other evaluation features, for example, 
care in data collection, related to the evaluation’s 
likelihood of uncovering an effect” (p. 97). 

418. Sheldon and Hopkins (2008). 



Other sources provided guidance for how 
and when to conduct more formal perfor- 
mance evaluations. Fullan (2001) suggests 
that two to three years are necessary for 
programs to begin to see representative 
positive effects, based on his research of 
school reform interventions.^^® Ross, Pot- 
ter, and Harmon (2006) provide guidance 
on the components of an effective evalu- 
ation and monitoring system for SES pro- 
grams; the panel believes that this guid- 
ance is applicable to other academically 
focused OST programs. Many publicly 
available resources that offer tools for as- 
sessing and monitoring program quality, 
either through observations or surveys, 
also are available.'*^^ 



419. Fullan (2001) is a research synthesis to 
which WWC standards are not applicable. 

420. Ross, Potter, and Harmon (2006) does not 
look at the effectiveness of any programs. 

421. For just two examples of what is available, 
see Yohalem et al. (2009) or the Bureau of Public 
School Options, Florida Department of Educa- 
tion (2007, n.d.). 
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